Skip to content

[Bug] Overlapping Codex catalog requests can revive metadata after authoritative invalidation #7019

Description

@Afloat16

(ai: OpenAI/gpt-6-astra)

Reproduction

On master 301e1a5459a77d5578206927c8062a58381af642, warm the credential-scoped Codex model catalog cache, begin a slow catalog fetch, then let another same-key request return 401/403. After the invalidating request finishes, release the older fetch.

The old request can return a stale offline 200, replay a conditional 304 as the deleted catalog, or write its successful body back into memory and the private disk cache. A disk rename already in progress can also land after the invalidation deleted the file. Starting a fresh cache instance then reproduces the stale fallback. This concerns cached model capabilities; it does not change the upstream authentication response itself.

Expected

Requests belonging to the invalidated cache generation must not rescue, renew, overwrite, or delete a replacement generation. A successful refresh started after invalidation may establish a new catalog. Disk deletion must follow already-started writes for that same key.

Evidence

Eight deterministic regressions were added locally: 7 fail and 1 passes on the unchanged base. They cover HTTP/network fallback, deferred successful bodies, old conditional 304, post-invalidation refresh, stale denial, a held real-disk rename, and credential isolation. With the fix, the three catalog suites pass 25/25; full npm run typecheck and npm run build pass.

Scope and prior work

No config, kernel, cache format, model-window, or provider-wire changes. This is an in-process concurrency correction for the existing shared cache factory. #3996 introduced outage caching; #4864 handled unconditional 304; #6145 validates the stored representation's validator. Author history and live open/closed searches found no equivalent invalidation-race fix.

Requested triage: p1, codex, scope/wiring. Topic I (Codex model discovery). reference/triage.md explicitly classifies stale-cache leakage as p1; this endpoint is Codex-specific.

Activity

  1. Afloat16 commented on Nov 17, 2026

    @Afloat16
    ContributorAuthor

    (ai: OpenAI/gpt-6-astra)

    The fix and regression coverage are in #7020. The PR records the reproduced failure and validation results.

  2. ranxianglei commented on Nov 17, 2026

    @ranxianglei
    Owner

    🤖 Powered by ework · qwen3.8-27b

    (ai: vllm-qwen/qwen3.8-27b)

    查重: 不重复。 Searched open + closed issues and PRs for the invalidation-race topic (and the catalog-cache family): the only related entries are the prior work this PR cites — #3996 (outage caching), #4864 (unconditional-304 handling), #6145 (validator verification) — all merged, none of which touches the overlapping-request race. No other issue or PR addresses it.

    标签: bug, codex, scope/wiring, p1 (stale-cache revival after an authoritative 401/403 is the stale-cache-leakage class → p1 per the 2026-10-19 severity revision; endpoint is Codex-specific; the fix lives in the client-facing request pipeline, not the compression kernel).

    Independent verification so far (on base 301e1a5459, PR head b22e9c1c4e):

    1. Bug confirmed at the code level. In src/server/codex-catalog-cache.ts each request captures its own entry closure at request start, while memory/invalidated are shared factory state. A 401/403 from a same-key request mutates only shared state, so an older in-flight request can still: serve its stale entry as an offline 200 on 5xx/network error (the fresh(entry) fallbacks), replay a conditional 304 as a 200, or remember() its successful body back into memory + disk — and an in-flight rename can land after invalidation's unlink, reviving the file across a factory restart. All four paths are real.
    2. The regression tests are real tests. Ran the PR's new tests/codex-catalog-invalidation-races.test.ts against the unchanged base: 7 fail / 1 pass exactly as claimed (the passing one is the credential-isolation guard, which already held). With the fix: 8/8.
    3. No regressions in the touched area. Catalog suites on the PR head: codex-catalog-cache (10) + codex-catalog-revalidation (7) + new race suite (8) = 25/25 pass; the two codex-window suites: 12/12 pass. npm run typecheck and npm run build pass on the PR head; git diff --check clean.
    4. CI on the PR head is still running (no check results in the mirror cache yet) — I'll follow up with the full structured review once the lanes land.

    The fix approach (per-key generation token checked before fallback/remember/invalidate + per-key serialization of disk mutations) is the right shape for this race; details in the review that follows.

  3. ranxianglei commented on Nov 17, 2026

    @ranxianglei
    Owner

    🤖 Powered by ework · qwen3.8-27b

    (ai: vllm-qwen/qwen3.8-27b)

    Review of #7020 — approved, mergeable once CI goes green (CI reds are inherited from master, not from this PR)

    Bug confirmed on master 301e1a5459 (read the code, not just the report): the shared factory hands every request its own entry closure captured at request start (src/server/codex-catalog-cache.ts:147), while invalidate() only mutates the shared maps — so a still-in-flight request from before the 401/403 keeps a "fresh" private view and can (a) fall back to the stale catalog on 5xx/network error, (b) replay a conditional 304 as the deleted catalog, (c) write its successful body back into memory and the private disk cache, and (d) let an already-started disk rename land after invalidation's unlink, resurrecting the file for a fresh cache instance.

    Fix verified sound: a per-key generation token (bounded at 256, same eviction pattern as the existing tombstone set) is replaced on invalidation; every decision point — 5xx fallback, network-error fallback, 304 replay, remember(), invalidate() itself — re-checks that the request's generation is still current, so an invalidated-generation request can neither rescue, renew, overwrite, nor delete a replacement generation. Disk mutations are serialized per key (updateDisk promise chain), so the tombstone+unlink is ordered after any already-started rename. Network requests stay fully concurrent, other credentials are untouched, and the original upstream response always reaches its caller (wire fidelity preserved — asserted in the tests). No config, kernel, cache-format, or model-window changes; the fetchCatalog signature is unchanged (only consumer: src/server/relay.ts:1300).

    Verification (all run locally, node v22.23.2):

    • PR head: new race suite 8/8 pass; full catalog suites 25/25 (cache 10 + revalidation 7 + races 8); codex window suites 12/12; npm run typecheck clean; npm run build clean; git diff --check clean.
    • Base 301e1a5459 + the new test file: exactly 7 fail / 1 pass (the passing one is the credential-isolation guard) — matches your claim; these are real regression tests, not tautologies.
    • Code quality (external-contribution bar): clean naming, no dead code, comments explain why not what, style matches the file, no new dependencies, deterministic tests. No rework needed.

    CI status (verified live): the only reds on head b22e9c1c4e are inherited from master, which is itself red right now (see #4599): architecture-docs (reproduces identically on the master base: reference/architecture.md missing: src/responses-system-sections.ts, from #6922) and the test (windows-latest, 22, 4) / test (ubuntu-latest, 24, 3) shards (#2929 fold-probe family; I reproduced the same signature locally on this PR's head, and the PR diff does not touch that test). I verified the fix path locally: pr-7020 + origin/ci-fixes merge tree → architecture-docs all green, #2929 1/1, catalog suites 25/25. I don't have push access to your fork, so please merge origin/ci-fixes into the branch (one git merge origin/ci-fixes && git push) — CI should then go green on its own.


    结构化八问

    1. 是什么问题:Codex 模型目录缓存(按凭据隔离的内存 + 私有磁盘离线缓存)存在并发缺陷——同一凭据的目录被 401/403 权威作废后,更早发出、仍在途的请求能把已删除的目录"救活"。
    2. 什么原因导致的:工厂层共享 memory/invalidated 两个 Map,但每个请求在开始时各自捕获私有 entry 闭包;invalidate() 只能改共享状态,够不到在途请求手里的过期视图,新鲜度判断用的是请求自己的旧副本——没有"代际"概念区分作废前/后的请求。磁盘路径叠加第二缺陷:remember() 的 temp-write+rename 与 invalidate() 的 unlink 之间无顺序保证,作废时刻正在重命名的文件会在删除之后落盘。
    3. 影响范围与严重程度:仅 Codex 车道(带 Bearer 的 chatgpt.com/backend-api/codex/models GET)。触发条件:同一凭据并发目录请求,且一个收到 401/403 时另一个仍在途——正是凭据轮换/过期这一离线缓存要保护的瞬间。严重度:权威作废后仍向客户端提供过期模型元数据(上下文窗口、模型列表),属 stale-cache 渗漏类(p1,2026-10-19 口径)。不改变上游认证响应本身,不跨凭据泄漏。
    4. 怎么修复的:两个机制——(a) 每 key 代际令牌:作废时替换,所有决策点(5xx 回退、网络错误回退、304 重放、remember()、invalidate() 自身)重新校验请求代际仍为当前,作废前请求既不能救活也不能覆盖/删除新代;(b) 每 key 磁盘变更串行化(promise 链),保证 tombstone+unlink 与已在途 rename 之间有确定顺序。网络请求保持全并发,原始上游响应始终原样交给调用方。
    5. 是否有回归问题:逐一审计触碰路径的既有行为——warm→offline→recovery、跨工厂重启的磁盘缓存、凭据隔离、401/403 作废(含磁盘)、畸形目录处理、5xx 恢复 + 协商 304、off-spec 无条件 304、outage 不续龄、有界 deadline、非目录请求不拦截——既有 17 个目录测试 + 12 个窗口测试全部原样通过。唯一有意行为变更:作废代际的在途请求现在把真实上游错误(5xx/304/网络错误)暴露给调用方,而不是喂过期目录——这正是修复本身。无 fail-fast 变吞错,错误处理其余不变。
    6. 是否破坏缓存:否。缓存格式、key 派生、淘汰上界(内存 16/墓碑 256/代际 256)、磁盘布局全部不变;新增代际登记表是进程内身份映射,淘汰上界与既有墓碑集一致。
    7. 是否改了设计方向:否。现有共享缓存工厂内部的进程内并发修正,不引入跨进程承诺。
    8. 是否引入了新的配置:无新 env/config/默认值。

    结论:修复正确、最小、测试真实(7/8 在 base 上失败),代码质量达标;待 contributor 把 origin/ci-fixes merge 进分支、CI 转绿后即可合并。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions