Skip to content

[PR] fix: preserve Codex model capabilities through catalog outages - #3996

Closed
zhou1736948757-cpu wants to merge 2 commits into
ranxianglei:masterfrom
zhou1736948757-cpu:2026-10-20_fix-codex-catalog-outage
Closed

zhou1736948757-cpu wants to merge 2 commits into
ranxianglei:masterfrom
zhou1736948757-cpu:2026-10-20_fix-codex-catalog-outage

Conversation

@zhou1736948757-cpu

@zhou1736948757-cpu zhou1736948757-cpu commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes #3993.

Preserve the last valid complete ChatGPT model catalog through temporary discovery outages, so Codex does not silently lose known context/tool capabilities and expose a very large eager tool catalog. Ported to native TypeScript on master 860e617 (package 0.1.202); the reporter's installed 0.1.147 local hook has already been verified separately.

This is a separate incident from the deferred-tool accounting fix in #2398 (already merged). Here the failed request had no deferred catalog to discount: catalog=visible ~249491, loaded=0. The supported boundary is model discovery, not subtracting eager tool definitions or increasing the context window.

Incident and upstream-client boundary

  • Codex 0.163.0 logged Unknown model gpt-6.1-sol is used. This will use fallback model metadata. after nearby catalog reset/cancellation.
  • Effective client window changed 475000 -> 258400 while the eager tool estimate became ~249491; preflight estimated ~286832 against its then-reserved 204000 input budget and had zero compressible ranges.
  • After recovery a different request used catalog898 / loaded22742; upstream actual input was 62552, then 57120. These are different requests; the failed request's actual usage was never measured and its original body was not captured.
  • Codex rust-v0.163.0 explicit-provider catalog error handling clears same-identity metadata, and its generic metadata lacks Code Mode/search capabilities. The incident cache-clear itself was not logged. Source evidence plus warning/window/tool transitions support the mechanism, rather than prove every causal step from an exact wire capture.
  • The ideal client fix is preserving known same-identity capabilities across transient errors (or refusing unsafe generic fallback). This PR changes only Bili's model metadata relay; it does not change Codex cache policy.

Implementation

  • Intercept only authenticated GETs to exact HTTPS chatgpt.com /backend-api/codex/models; inference POSTs and other providers remain on the existing transport path. This metadata side request is independent of plugin/proxy compression mode.
  • Retain exact validated nonempty full JSON catalog; do not synthesize model IDs, windows, instructions, tool schemas or capabilities.
  • SHA-256 identity key includes full URL/query/client version, Bearer token, account, organization and project. Raw credentials/account IDs are not persisted in cache records or logs.
  • Private existing-XDG cache directory, atomic file replacement, 1 MiB/catalog, 16 entries, six-hour maximum stale age. Successful online discovery wins; offline hits never extend age. Same-identity 304 revalidation supplies a full 200 catalog.
  • Cover transport failures and HTTP500/502/503/504, with four-second network/body deadline only when a usable cache exists. Caller cancellation stays cancellation. Authentication/other explicit errors and malformed/empty catalog successes invalidate old metadata. Cold, expired, corrupt and credential/version-mismatched caches require online success.
  • Model discovery can reach its cache even when the upstream host circuit is open. An offline replay does not record network recovery, clear alerts, or clear the inference circuit.
  • English/Chinese client docs and architecture map updated.

Validation

  • npx tsc --noEmit --project tsconfig.json: passed.
  • npm run build: passed, including dist import annotations and external-zod gates.
  • node scripts/check-unused-exports.mjs: clean (five existing waivers).
  • 33 targeted tests passed: nine new cache tests plus side-request isolation, cancellation lifetime, idle timeout, upstream timeout and circuit regressions.
  • Actual installed Codex CLI 0.163.0 app-server through native Bili/local HTTP metadata fixture: online -> HTTP503 -> socket disconnect -> recovered, passed. Each phase deletes only the temporary Codex metadata cache, forcing genuine model discovery; checks model availability, Code Mode/search/window metadata and absence of the unknown-model warning. Synthetic OAuth only; no inference requests or real auth/history attached.
  • Reproduce native fixture (opt-in): ACP_TEST_E2E_CODEX_CATALOG=1 E2E_CODEX_BIN=/path/to/codex node --import tsx --test tests/e2e/e2e-codex-catalog.test.ts.
  • The full-suite verdict is pending upstream CI on this pushed head, per repository policy. No real-provider inference/lifecycle suite was run; this PR is not merged.
  • Initial local test-runner setup used a Windows absolute path for NODE_OPTIONS and failed before running tests; corrected to a file URL. The successful 33-test run uses an external test-only preloader to disable ambient Windows proxy discovery and isolated XDG directories; no registry/system proxy settings were modified.

Config surface and review

No new config keys, CLI flags, production environment variables, model-window changes, version bumps or kernel/pi-subagents edits. Internal policy limits are explicit above. The six-hour bounded stale-capability tradeoff and credential-scoped persistence require human review; do not auto-merge. An account/credential rotation intentionally produces a cold miss. The protection needs a prior successful catalog fetch and cannot make inference work while the upstream network is down.

Initial CI status

The initial lifecycle-ledger-guard failure is reproduced on unchanged base 860e617: 148 readers / 128 active ledger entries / 16 identifiers exempt, with the same 22 missing registrations. This PR introduces no BILI_* production environment reads. The existing upstream issue #3994 and PR #3995 already track the baseline repair; their policy registrations are intentionally not duplicated in this fix. Other checks remain in progress/queued at submission.

@ranxianglei ranxianglei mentioned this pull request Oct 20, 2026
@ranxianglei ranxianglei changed the title fix: preserve Codex model capabilities through catalog outages [PR] fix: preserve Codex model capabilities through catalog outages Oct 20, 2026
@zhou1736948757-cpu

Copy link
Copy Markdown
Contributor Author

Initial CI triage: lifecycle-ledger-guard is already failing on unchanged master 860e617 with exactly the same 22 missing registrations (148 readers, 128 active entries, 16 exempt). Reproduced from a clean archived base source tree. This fix adds no production BILI_* reads. Existing issue #3994 / repair PR #3995 cover the baseline failure; keeping that registration repair separate. Local typecheck/build and 33 targeted tests plus the real Codex four-phase catalog fixture passed.

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重:不重复——open+closed issue/PR 检索「Codex catalog 瞬断 → fallback 元数据降级」主题无匹配;#3994/#3995 是 lifecycle-ledger 修复,机制不同(CI 注册台账),仅作基线红背景相关。
标签:bug + codex + scope/wiring + p2(功能缺陷但有绕行——上游恢复即自愈、无数据丢失;Codex 车道;请求管道接线层)。

独立审核完成,以下全部在 PR head faf6e97b 上亲自验证,未照抄原帖结论。

独立验证证据

结构化八问

  1. 是什么问题:Codex 模型目录 API(chatgpt.com/backend-api/codex/models)一次瞬断后,Codex 丢弃同身份缓存元数据、退回 generic metadata——丢 Code Mode/search tool 支持、上下文窗口从 475K 缩到约 258K,长会话随即 preflight 超预算且无可压区间,上游恢复前不可用。
  2. 什么原因导致的:根因在 Codex 自身的处理(显式 provider catalog 请求失败即清缓存用 generic fallback);bili 改不了 Codex 内部,但代理正好站在这条请求路径上——坏状态的生产者是「代理把错误/空响应交给 Codex」,本 PR 在源头修:瞬断时回最后一次已验证的完整 catalog 逐字字节,让客户端根本不进 fallback。
  3. 影响范围与严重程度:仅 Codex 车道;触发条件 = 该 GET 瞬断(传输中断/5xx/挂起)且存在 6h 内有效缓存;故障窗口内会话降级(能力丢失 + 可能 preflight 溢出),恢复后自愈,无数据丢失 → p2。
  4. 怎么修复的:代理层持有有界的「同凭据最后一次有效完整 catalog」缓存(6h 过期 / 单条 1MiB / 最多 16 条 / 有缓存时 4s 截止);字节精确保存,只在瞬断时回放,offline 命中不续期;401/403/其他显式错误或畸形 catalog 主动失效(磁盘 tombstone 防删除失败复活);隔离键 = URL+token+account/org/project 指纹哈希,不落盘原始凭据;冷启动/过期/不匹配仍要求在线成功,诚实失败。
  5. 是否有回归问题:逐项清点既有行为——非 catalog 请求零变化(method+origin+path+Bearer 四重门、短路求值,热路径成本≈一次 URL 解析);可达性记录/告警清除/circuit 成功记录仅对 offline 回放跳过,真实 200 照常记录;circuit open 态快速失败对所有非 catalog 请求保持,catalog GET 获得有界在线机会(有缓存 4s 封顶、无缓存走常规超时,无缓存则诚实失败);304→200 全量转换对消费者语义等价;offline 回放新增一个 x-bili-model-catalog 响应头(客户端忽略未知头);两种压缩模式均不受影响(URL 级匹配,与 pluginMode/proxyMode 无关)。未发现无意回归。
  6. 是否破坏缓存:不破坏任何既有缓存/重试/游标/去重策略;新增一个独立磁盘缓存(用户缓存目录下 codex-model-catalog-v1 子目录,0o600 原子写),与既有缓存无交集。
  7. 是否改了设计方向:不转向——additive 模块(一个新文件 + relay.ts 17 行接线),架构/边界/数据流不变。
  8. 是否引入新配置:生产侧无(无新 env/config 字段/默认值);测试侧新增门控变量 ACP_TEST_E2E_CODEX_CATALOG(+E2E_CODEX_BIN),沿用既有 ACP_TEST_* e2e 惯例,默认 skip。

发现(均不阻塞,按规则列出;本线程是 [PR] 自动镜像,按 loop guard 写评论等 owner 定夺,不单开 issue)

  1. 真客户端 e2e fixture 与 codex 版本 schema 绑定:本地 codex-cli 0.147.0 要求 catalog 含 supports_parallel_tool_calls 字段,fixture 没有 → online 阶段直接假红(补上该字段后整套 6.8s 通过,证明机制本身端到端成立)。fixture 贴合作者本地版本(~0.163),换版本跑会出看似回归的假红。建议测试头注明适配的 codex 版本范围或扩充 fixture 字段集。
  2. 新 e2e 无 CI 接线:workflows/run-ci-tests 均未引用该门控变量,套件不会自动跑(其他真客户端车道在 CI 里跑)。是否接 CI 由你定;若接,先解决发现 1。
  3. 429 也触发失效:符合 issue 规格原文「other explicit errors」,但限流≠catalog 变更,限流期间失效恰好丢掉保护。属已批准设计,建议保持现状,此处备案。
  4. ECONNABORTED 不在瞬断码表:undici 通常把 socket 中断映射为 UND_ERR_SOCKET/ECONNRESET,实际影响很小,如线上遇到再补。

CI 状态:PR head 有 4 个 test shard 红;master head(56a0f5a46)同样有 5 个 test shard 红——test 车道基线红风暴早于本 PR(另有 PR 在修 CI),两头红的分布一致指向存量红而非本 PR 引入;本审核环境取不到 CI 日志,无法点名具体测试,但 guard/static/hermetic e2e 各车道在 PR head 均绿。

判定:代码质量高(diff 干净、双向测试、en/zh 文档同步,无需对齐重构);方向正确(疏不堵,修在坏状态生产者);未发现阻塞缺陷。按 §7.4,本 PR 触碰 wire 形状(新响应头、304→200 转换)+ fallback 行为(stale capability 最长 6h)+ 安全相关(凭据指纹磁盘持久化、circuit 快速失败旁路)→ 需人工 review。建议 rebase 最新 master、等 test 车道转绿后合并。

@ranxianglei

Copy link
Copy Markdown
Owner

(ai: zhipuai-lb/glm-5.3)

已并入合并列车 #4525(master 6d7c587,合并方式 merge commit,提交保留原作者署名)。本 PR 关单收尾;若列车中对应提交出问题,回滚点为 #4525 单提交 revert。

感谢 @zhou1736948757-cpu 的贡献——你的提交以原作者署名随列车进入 master(合并方式为 merge commit),仓库贡献者列表可见。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working codex p2 scope/wiring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug][Codex] transient model-catalog outage triggers generic fallback and tool-context explosion

2 participants