Skip to content

[PR] Fix permanent unified-shadow DIGEST DRIFT on ref-floored sessions (Codex Desktop): replay leg minted a phantom byRef entry - #5597

Open
ranxianglei wants to merge 9 commits into
masterfrom
2026-11-03_ref-floor-digest-parity
Open

ranxianglei wants to merge 9 commits into
masterfrom
2026-11-03_ref-floor-digest-parity

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Nov 3, 2026 •

Copy link
Copy Markdown
Owner

What this fixes

Field report #5589 — v0.1.212, Codex Desktop app-server 0.166.0, long-running conversation:

[storage] unified shadow DIGEST DRIFT
unified replay != legacy load view [diverged: state; state: messageRefs]
dropped the ingest watermark, the next save re-ingests the full tail

The same line repeated on every sampled verify of that session's lifetime.

Follow-up field data in the same thread (69 deduped drift events across 6 sessions, 4 frozen envelopes) splits into three signatures: 63x [diverged: state; state: messageRefs] (facet A below), 4x [diverged: blockContents] on one session (facet C below), and 2 isolated single-event lastMessages tail-slot byte deltas that self-healed on the next sample (the documented delta-fast-path trade-off for host-side rewrites below the watermark — no action). This PR fixes both structural facets; each is its own labeled commit. The report's second symptom (snapshot-shadow lost-tail:N) is a separate observability lane that legitimately flags client-side history rewrites — see the issue thread; no change here for it.

Facet A mechanism — ref-floor phantom byRef entry

  1. reconcileNativeCompactionBoundary ([Bug] #2436 的宿主本地压缩检测只接在 OpenAI 车道,Anthropic 车道缺失——未宣告宿主压缩后 nudge 广播的 compress ref 永久不可解析 #2592/F5) pins the session's ref high-water mark after every Responses/codex native compaction: byRaw["bili:ref-floor"] = mNNNNN with no byRef counterpart — a documented invariant (the floor only exists to hold the kernel cursor high).
  2. finishPlan persists EVERY byRaw entry as a refs row (src/storage/ingest.ts).
  3. LegacyViewBuilder (the replay leg) inverted every non-BLOCKED row into byRef — including the floor row — minting a phantom mNNNNN → "bili:ref-floor" entry the live map never has.
  4. [PR] fix(storage): digest object keys in sorted order — single-instance shadow drift (#4392) #4685 (shipped in v0.1.209+) made the parity digest order-insensitive, so this content delta became the only remaining divergence dimension: [diverged: state; state: messageRefs] on every sampled verify (every 20 ingests) → watermark drop + full-tail re-ingest each time → after three consecutive samples the session's JSON envelope writes are frozen for the rest of the process lifetime — permanently, because the floor never leaves byRaw while the drift can never be clean. In shadow mode (the default) the JSON envelope is the primary durable record, so a restart after freeze resumes from a stale snapshot (data-loss class).

Why Codex specifically: codex/Responses native compaction takes the REBASE-type boundary path, which pins unconditionally; the anthropic-wire /compact path (dsh/opencode/pi lanes) only pins when an archive prune frees the max number. dsh lanes therefore rarely pin; long-running Codex Desktop sessions almost certainly do.

Behavior change (old → new)

  • Old: any stored session whose byRaw contained bili:ref-floor drifted on every sampled shadow verify → repeated watermark drops + full-tail re-ingests → after three consecutive samples its JSON envelope writes froze for the whole process lifetime. Full-mode boot additionally resolved citations of the floor number to the synthetic id instead of falling through to tombstone handling.
  • New: the replay reconstruction skips the floor entry exactly like BLOCKED. Shadow verify of a floor-pinned session now passes; a genuine divergence around the same number still drifts (negative-gate test). byRaw keeps the row — the cursor high-water must survive full-mode boot, so ref allocation is unchanged.
  • Heal: no data repair needed — the stored rows were always correct; only the in-memory reconstruction was wrong. Affected users clear the condition by upgrading and restarting bili (the per-process frozen set resets; the first save verifies clean). Turns saved only in memory during an already-frozen period were lost at the last pre-upgrade restart — nothing more will be lost after.

Facet C — orphan-journal ghost latch ([diverged: blockContents])

Field evidence: one session (S6), 4 events, escalated to envelope freeze. Every stored block row postdates the first drift and no compaction/rebase line precedes it within the retained log window — the divergence came from state carried over from an earlier session generation, not from a visible compaction at drift time.

Mechanism:

  1. An earlier generation leaves orphan contents — the [Bug][proxy] unframed host-side history deletion leaves zombie fold blocks rendering carriers forever — one carrier flaps in/out every turn (persistent prefix-cache churn) #2695 reap shape: a block is removed from state.blocks while its original text stays in blockContents (kept for decompress). finishPlan journals those entries into sessions.orphans_json.
  2. A native-compaction REBASE ([Bug] #2436 的宿主本地压缩检测只接在 OpenAI 车道,Anthropic 车道缺失——未宣告宿主压缩后 nudge 广播的 compress ref 永久不可解析 #2592/F5, the Codex path) later clears blockContents in place — an authoritative clear; the plan then carries an empty orphan set (null).
  3. The sqlite store applied MERGE semantics to the journal: null meant "keep whatever is already stored", so the pre-clear ghosts stayed in orphans_json forever.
  4. The sampled verifier compares every journal id against the live view; a ghost id can never match → [diverged: blockContents] on every sampled verify → streak escalation → envelope freeze — permanently, for the same reason as facet A: no future save can ever make the sample clean again.

Fix = per-save authoritative replacement: a populated set replaces the journal, an empty set resets it. Safe because a save can only ever see the full set or a genuine clear — LazyBlockContents re-materializes from disk on iteration or throws refusing a thinned view, so the "trimmed live map" state the old merge semantics protected is unreachable in production. The mysql upsert had the same latent class one notch worse (it never wrote orphans_json at all); it now carries the column.

Behavior change (old → new):

  • Old: pre-clear orphan entries survived every later save; any session that reaped blocks and later hit a native-compaction rebase latched a permanent [diverged: blockContents] drift + envelope freeze.
  • New: the journal follows the live set per save; after an authoritative clear the ghosts are gone and sampled verifies pass. Orphan content the live side still carries is unaffected (pinned by the updated [Bug][storage] Retained orphan originals freeze unified session ingestion; cold loads restore hours-old state #4216 test).
  • Pinned-test expectation updated: tests/storage-orphan-representable.test.ts step 4 previously asserted "a save whose live map was trimmed keeps the journal"; that input shape is unreachable in production (see above) and the assertion now pins the reset invariant instead. Disclosed per the behavior-change rule.

Heal: same as facet A — upgrade + restart (per-process frozen set resets; first verified-clean save resumes envelope writes). No DB repair needed: the journal rows were always a faithful copy of what the store was told; only the retention policy was wrong.

Config surface

None. No new env/config/CLI surface.

Kernel

Untouched (no kernel/** changes).

Tests

  • New tests/issue5589-ref-floor-parity.test.ts: e2e red/green (the pre-fix run reproduces the exact field log line byte-for-byte, incl. [v=0.1.212] [diverged: state; state: messageRefs]), shape assertion (floor survives the round-trip in byRaw, no phantom byRef), negative gate (a genuine divergence around the floor number still drifts — After idling for one hour, the fold override is lost: the client resends byte-for-byte identical content, and the previo | 闲置一小时后折叠覆盖丢失:客户端重发逐字节不变,已折叠的 ~240K token 内容自己回到上下文 #4323 heal direction preserved).
  • New tests/issue5589-orphan-journal-ghost.test.ts (facet C): hermetic red/green — the pre-fix run reproduces the exact field log signature [diverged: blockContents] — dropped the ingest watermark…, plus a negative control proving a persisting orphan does NOT trip the verifier.
  • Adjacent targeted suites (issue4392-canonical-key-order, issue4986-digest-drift-ref-order, issue4323-drift-escalation, issue3919-shadow-drift-forensics, issue3407-shadow-blockcontents-order, issue5412-async-drift-latch-parity, storage-orphan-representable, storage-reap-journal, storage-orphan-recovery, storage-json-writer-off, snapshot-shadow, storage-batched-verification, issue2592-rebase-floor, claude-transcript-lineage): 69/69 pass. Typecheck clean. Full-suite verdict = CI on this head.
  • §7.4 note: touches message-ref/id logic (replay-leg reconstruction) → human-review tier, not auto-merge.

Third commit: nested field drill-down in the drift diagnosis (#5589 field evidence, v0.1.215)

A follow-up field report on v0.1.215 (Codex CLI 0.169.0) showed 7× unified shadow DIGEST DRIFT all on diverged: lastMessages — 3× with the SAME id/role/text but a constant +26 canonical bytes inside rawResponsesItem (535→561, 592→618, 729→755). The #5677 top-level field diff names the sidecar (fields: rawResponsesItem) but not WHICH wire member moved, so the exact trigger stayed unidentifiable from the log.

This commit drills one level: for each divergent top-level field whose value is a plain object on both legs, it names the inner members whose canonicalized values disagree. Wire schema names only — values never reach the log line (same privacy contract as the #5677 field names). Bounded: at most 5 inner members per field, sorted. Fields absent on one leg are not drilled (the top-level name already says "the sidecar appeared/disappeared"); arrays are not drilled (index-level diffs are a different shape). Wired into both call sites (header-level describeViewDivergence + the per-row compareMessages that produces the sampled verify line).

⚠️ Log-line format change (disclosed per owner policy 2026-09-22): the drift diagnosis line gains an ADDITIVE segment when an object-typed field diverges. The line prefix stays byte-identical (existing greps keep working); only the new segment is appended:

  • old: ...; fields: rawResponsesItem
  • new: ...; fields: rawResponsesItem; rawResponsesItem: summary

Diagnostic only — no behavior, wire, storage, or config change. The 3× +26 shape is the documented #4614 slot-rewrite class (self-healing: watermark drop → full re-ingest → converge; the report showed no escalation/freeze). This commit makes the next occurrence name the exact member instead of leaving it to inference.

  • New test in tests/issue5677-drift-field-diag.test.ts (7 cases: added inner member / changed value / multiple sorted / absent-on-one-leg no-drill / >5 capped / array not drilled / non-object no empty segment). Targeted suite 4/4 pass; adjacent shadow-drift suites (issue5408, issue3919, fold-round2) 11/11 pass. Typecheck clean. arch-docs guard green.

Closes #5589

After a Responses/codex native-compaction rebase (#2592/F5) pins the
session's ref high-water mark, byRaw carries the synthetic
"bili:ref-floor" entry with no byRef counterpart (documented invariant).
The refs ledger persists every byRaw entry as a row, and LegacyViewBuilder
inverted every non-BLOCKED row into byRef — including the floor row —
minting a phantom mNNNNN→floor entry the live map never has. Since #4685
made the parity digest order-insensitive, that content delta was the last
divergent dimension left: DIGEST DRIFT [diverged: state; state:
messageRefs] on every sampled verify of any session that ever pinned a
floor, watermark drop + full-tail re-ingest each time, escalating to
frozen JSON envelope writes after three samples — permanently, because
the floor never leaves byRaw while the drift can never be clean. In
shadow mode the JSON envelope is the primary durable record, so
restarting after a freeze resumes from a stale snapshot. Codex Desktop
sessions pin the floor on every native compaction (rebase-type boundary),
which matches the field report: v0.1.212, long-running Codex
conversation, persistent drift.

The replay reconstruction now skips the floor entry exactly like BLOCKED:
byRaw keeps the row (the cursor high-water must survive full-mode boot);
byRef stays the strict inverse of real messages only. REF_FLOOR_RAW_ID
moved to state-hydration.ts so the storage seam can consume it without
importing the session layer; session.ts re-exports it for existing
importers.

Tests: tests/issue5589-ref-floor-parity.test.ts — red run reproduces the
exact field log line byte-for-byte; negative gate proves a genuine
divergence around the floor number still drifts. Adjacent suites 50/50.
@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-11-03_ref-floor-digest-parity (a5b269b)

Latest build — install this exact version

npm install -g billion-context@0.0.10658-pr.5597.e8968d7

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
a5b269b billion-context@0.0.10658-pr.5597.e8968d7 build
9a558e7 billion-context@0.0.9177-pr.5597.97a6fb4 build
2aeac3f billion-context@0.0.8267-pr.5597.321c04e build
a2e0af4 billion-context@0.0.8266-pr.5597.4b9d868 build
e60f2e4 billion-context@0.0.8257-pr.5597.0efbd98 build
6b0d37b billion-context@0.0.8254-pr.5597.e34e8dd build
81b36c8 billion-context@0.0.7984-pr.5597.7c22066 build
d265b35 billion-context@0.0.7977-pr.5597.d60d067 build
86751d7 billion-context@0.0.7971-pr.5597.e6b8a1c build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr5597.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei ranxianglei changed the title Fix permanent unified-shadow DIGEST DRIFT on ref-floored sessions (Codex Desktop): replay leg minted a phantom byRef entry [PR] Fix permanent unified-shadow DIGEST DRIFT on ref-floored sessions (Codex Desktop): replay leg minted a phantom byRef entry Nov 3, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复——本 PR 即 #5589 的修复(唯一在途针对该机制的 PR);#5106 是另一观测车道(snapshot-shadow lost-tail),机制不同、本 PR 不涉。
标签: bug / codex / scope/storage / p1(p1=数据丢失类:envelope 冻结后重启恢复陈旧快照)。

① 是什么问题:unified shadow(默认模式)下,凡走过 Codex/Responses native compaction 的长会话,每次抽样校验必报 DIGEST DRIFT [diverged: state; state: messageRefs]——水位反复丢弃、全尾重灌;连续 3 次抽样后本进程对该会话的 JSON envelope 写入永久冻结,重启从陈旧快照恢复。

② 什么原因导致:native compaction 清理旧 ref 时,游标高水位用合成键 byRaw["bili:ref-floor"]=mNNNNN 钉住——该条目没有 byRef 对应项(文档化不变量,floor 只压游标)。refs 账本把全部 byRaw 行持久化,而回放腿 LegacyViewBuilder 把每个非 BLOCKED 行反转入 byRef——floor 行被反转入内,铸出活态 map 从不存在的幻影条目。#4685 去掉 key-order 假阳性后,这是唯一残留分歧维 → 永久漂移。

③ 影响范围与严重程度:所有走过 native compaction 的 Codex/Responses 长会话(v0.1.212 字段实锤,Codex Desktop app-server 0.166.0);shadow 为默认模式,面宽。严重度 p1 数据丢失类:冻结期内仅内存保存的回合在重启时丢失。

④ 怎么修复:回放腿的唯一反转点跳过 floor 行,与 BLOCKED 同等待遇;byRaw 保留 floor 行(full-mode boot 的游标钉必须存活,ref 分配零变化)。无需数据修复:存储行一直正确,只有内存重建错;升级+重启即清(frozen set 进程级,首存即验证干净)。

⑤ 是否有回归问题:审过的既有行为——(a) BLOCKED 跳过路径不变(同条件并列);(b) byRaw 持久化/重建不变(floor 行照旧往返,cursor 分配零变化);(c) fork-adoption 两条自造 byRef 的路径(fork-adotion.ts:217/253)只遍历真实消息 id,合成键进不了其域,无需改——已核查;(d) 披露的行为变更:full-mode boot 对 floor 编号引用的解析,旧=解析到合成 id「bili:ref-floor」,新=fall through 到 tombstone/unknown——与活态行为一致(活态从未有该 byRef 条目),属回归对齐而非漂移;(e) kernel/ 未触碰、配置面未动、持久化格式未变、wire 协议未变。negative-gate 测试护住不过修:同一编号附近的真分歧仍报 drift(#4323 heal 路径完好)。无意外回归。

⑥ 是否破坏缓存:不涉及 prefix cache 语义(修的是存储校验腿)。反而消除了水位丢弃/全尾重灌的恶性循环——那是本 bug 的症状。

⑦ 是否改了设计方向:不变。REF_FLOOR_RAW_ID 常量搬家(session.ts→state-hydration.ts + re-export)为纯解耦(storage 缝不再依赖 session 层),语义逐字保留,既有测试 importer 经 re-export 不受影响。

⑧ 是否引入新配置:无。

本地定向验证(全量以 CI 为准):

CI 现状:d265b35a5 重跑中,19/30 已过、0 红(余为 test shard 运行中)。

判定:可合(待 CI 全绿确认)。§7.4:触碰 message-ref/id 逻辑 → 人工合并层,请 owner 合入。

@ranxianglei ranxianglei added bug Something isn't working codex p1 scope/storage labels Nov 3, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI 更新:d265b35a5 全绿(30/30,0 红)。审核结论维持可合,请 owner 合入。

@Maziar123

Copy link
Copy Markdown
Contributor

Independent reproduction on Node 26.13.0, using PR head d265b35a57 and current master d244ffff27f3:

This independently confirms the active v0.1.212 failure mode and the fix. It does not recover turns already lost before a restart; existing frozen in-memory sessions still need a safe upgrade/restart path.

@Maziar123

Copy link
Copy Markdown
Contributor

Current upstream rollup is not green: 58 success, 1 failure, 39 cancelled, 31 skipped. The sole failure is Windows Node 22 shard 3, #3380 P2 integration: failures escalate on the ladder, then heal clean: retryAt sits on the 80ms ladder step (got 25ms) at tests/issue3380-dsh-rereg-selflock.test.ts:285. That test is outside this PR's changed files. I am reporting the timing failure, not treating it as a pass or attributing it to this patch.

@ranxianglei ranxianglei left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(ai: vllm-qwen/qwen3.8-27b)

Review posted as COMMENT: GitHub blocks the account from formally approving its own PR (422 self-approval) — the verdict below stands as the review-lane approval.

(ai: vllm-qwen/qwen3.8-27b)

Verdict: APPROVE. Mechanism verified end-to-end against the code; the fix is the minimal correct one; the old→new disclosure matches what the diff actually does.

结构化八问

  1. 是什么问题:任何曾经钉过 ref floor 的会话(Codex/Responses native compaction)在每次抽样的 unified-shadow 校验上漂移([diverged: state; state: messageRefs])→ 每次丢水位 + 全尾重灌 → 连续三次抽样后该会话的 JSON 信封写入被冻结到进程生命结束。shadow 模式(默认)下,冻结后重启从过期快照恢复——数据丢失级。
  2. 什么原因导致的:#2592/F5 在 byRaw["bili:ref-floor"] = mNNNNN 上钉高水位且无 byRef 对应项(文档化不变量——把 kernel 游标压在已释放编号之上);finishPlan 把每个 byRaw 条目都持久化为 refs 行;LegacyViewBuilder 回放腿把每个非 BLOCKED 行反投影回 byRef——铸造了一个活地图从未有过的幻影 mNNNNN → "bili:ref-floor" 条目。#4685 让 parity digest 对顺序不敏感后,这个内容差成为唯一残留的漂移维度。根因类别:账本消费者漏掉了 BLOCKED 早已具备的同款跳过。
  3. 影响范围与严重程度:所有 floored 会话。Codex/Responses 的 REBASE 型边界无条件钉 floor(长时运行的 Codex Desktop 会话几乎必然命中——与现场报告 #5589 一致);anthropic-wire /compact(dsh/opencode/pi 车道)仅在归档剪枝释放最大编号时才钉,很少命中。严重程度 p1 级:每会话永久退化(信封冻结)+ 默认 shadow 模式下的过期恢复风险。
  4. 怎么修复的:回放腿构建 byRef 时像跳过 BLOCKED 一样跳过 floor 行(单一谓词,src/storage/ingest.ts:610)。常量移到 src/state-hydration.ts,让存储缝无需导入 session 层即可消费(session.ts 保留 re-export 给既有导入方)。行仍留在 byRaw——full-mode boot 依旧看到游标高水位,ref 分配不变。两条物化腿(shadow 回放与 full-mode boot 的 viewFor)走同一个 builder,一处修复覆盖两条腿。
  5. 是否有回归问题(既有行为逐项清点):
    • 全仓审计每个 byRaw/byRef 构造点:fork-adoption.planForkAdoption 遍历的是真实消息 id(block effectiveMessageIds / incoming ids),从不遍历 byRaw 键 → floor 进不来;HTTP fork 端点克隆父会话活 messageRefs(其 byRef 本就没有 floor)并用真实前缀消息重断言 → 干净;cloneStoreForRefs 操作 CCR content store(另一结构)→ 不涉及;BatchedViewVerification 复用 loadLegacyView(即修复路径本身);store 一致性检查只用正向投影(rawId→ref)→ 不涉及。回放腿是唯一的行→byRef 反投影点,与 PR 声明一致。
    • 负向门:围绕 floor 编号的真实分歧仍然漂移(新测试第 3 条)——#4323 的 heal 方向保住,修复不掩盖真漂移。
    • 游标语义不变:floor 行 round-trip 后仍在 byRaw(新测试第 2 条)——ref 分配逐字节等价。
    • 非阻塞观察(#2592/F5 起就存在,本修复使其暴露面缩小):getLoadHeader 的 mapped_refs SQL(src/storage/store.ts:2690)会把 floor 行计入,floored 会话比 loadResidentFields 漂移态仲裁用的信封 byRef 计数(src/persist.ts:1906)恒多 1;效果仅是「已漂移状态下平局时系统性偏向 indexed 视图」,未发现正确性破坏。在那里排除 raw_id='bili:ref-floor' 是一行谓词的事,但它改动加载路径仲裁输入——本 PR 不动,留给 owner 裁决。
  6. 是否破坏缓存:否。模型前缀缓存语义未触碰;变更移除的是重复的水位丢失/全尾重灌(存储侧重复劳动,不是模型缓存)。ref 分配不变。
  7. 是否改了设计方向:否——一个消费者对齐文档化不变量;常量搬家是依赖方向清理(存储层不应导入 session 层),行为中性。
  8. 是否引入新配置:无(已核实,PR 声明无配置面)。

范围说明:四个 golden 文件

tests/golden/storage-render/*.json 的更新(四条 wire 每 turn 统一 +49 bytes 的摘要位移)属于 #5555 false-advertising 文案清扫落 master 后未重生成 golden 的继承红(按 #4599 纪律收敛进本分支)。已验证幂等:在本 head 跑 scripts/update-storage-render-goldens.ts 产生零 diff——golden 与当前代码精确一致。

验证证据

  • 本地定向运行(d265b35a5):新测试文件 + 12 个相邻套件(drift-escalation、digest-drift-ref-order、canonical-key-order、rebase-floor、transcript-lineage、json-writer-off、snapshot-shadow、batched-verification、golden-storage-render 等)58/58 通过;typecheck 干净。
  • 报告者独立复现(真实工作站数据):五个冻结会话全部带 floor 行;master 代码复现精确漂移 + 幻影 byRef["m00042"];本实现三条回归全过。
  • CI(实质 head d265b35):guards 全绿,test 分片 10/11 绿;唯一失败 = test (windows-latest, 22, 3) 于 tests/issue3380-dsh-rereg-selflock.test.ts:285(时序断言,与本 diff 零交集,本地通过)——已在 #4599 报备,不归因于本补丁。评审顺手小修(re-export 注释订正)以 81b36c8 落分支,最终绿等其重跑。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Review 完成——APPROVE(完整八问结论见上方 review;GitHub 禁止账号自批自己的 PR,故以 comment-review 形式落定)。

@Maziar123 感谢独立 red/green 复现——与代码级审计逐条吻合:五个冻结会话全部带 floor 行、master 代码铸出幻影 byRef["m00042"]、修复后三条回归全过。你补充的边界也成立:冻结期内已丢的回合不可恢复,升级+重启只清场向前。

已顺手修:src/session.ts 的 REF_FLOOR_RAW_ID re-export 注释把 "fork-adoption" 列为既有导入方,但该文件在 PR 前后都未导入过此常量(实际只有 tests)——订正为 "(tests)" + commit 81b36c8(纯注释,零行为影响;顺带触发 CI 全量重跑取干净信号)。

CI 说明:唯一红分片 test (windows-latest, 22, 3)(issue3380-dsh-rereg-selflock 时序断言,80ms ladder 实得 25ms)与本 diff 零交集、本地 Linux 6/6 绿——已在 #4599 报备为疑似 Windows 时序 flake,不作为本补丁缺陷处理。

收工判定:可合并——等注释级新 head 81b36c8 的矩阵转绿即可。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Review lane 收工总结(详情见上方 review 与楼层 12):

等 CI 转绿后请 owner 合入。

ci-check added 3 commits November 5, 2026 20:27
…facet C)

A native-compaction rebase clears blockContents in place, but the store
kept whatever the sessions.orphans_json journal already held (merge
semantics): pre-clear entries became ghosts that the sampled verifier
flagged as [diverged: blockContents] on EVERY verify - a permanent drift
the escalation could never heal (field evidence S6 on 0.1.212).

Per-save replacement: null resets the journal, a populated set replaces
it. The old retention protected a production-unreachable state (a
thinned plain map) - LazyBlockContents re-materializes on iteration or
throws, so a save only ever sees the full set or a genuine clear; the
pinned #4216 expectation for that case is updated accordingly. The mysql
upsert now carries orphans_json (same latent class: it never wrote the
journal column at all).

Hermetic repro tests/issue5589-orphan-journal-ghost.test.ts reproduces
the byte-identical field log signature pre-fix, plus a negative control.
@ranxianglei
ranxianglei force-pushed the 2026-11-03_ref-floor-digest-parity branch from a2e0af4 to 2aeac3f Compare November 5, 2026 14:33
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

结论:本轮 10 个红的 test 分片已全部转绿。 这是继承性红(master born-red,非本 PR 缺陷);收敛提交在我接手前已在分支上,我沿上游 check-runs 直接确认了终态全绿。

head 变化:6b0d37bae(红)→ e60f2e48(绿)。当前 head 的 check-runs:37 success / 14 skipped(默认跳过的 e2e·real-client 门)/ 0 failed / 0 进行中。原 10 个红分片全部落在 success 之列。

结构化八问

  1. 是什么问题:跨 ubuntu/windows × node22/24 的 10 个 test 分片全红,报错集中在 dsh 车道回归测试——典型如 tests/issue2419-dsh-skill-catalog.test.ts:232,断言 catalog 消息带 BLOCKED ref(expected true / actual false)。
  2. 什么原因导致:master 侧 born-red,不是本 PR 引入。01fcc5d68(DSH: If the first request of a session originates from a bypass call, compression for the entire session is permanently | DSH: If the first request of a session originates from a bypass call, compress for the entire session is permanently inv | DSH:会话首个… #5656/[Bug][Codex Launcher] Marketplace Git repositories escape CODEX_HOME through Bili .tmp symlinks #5584,经 [PR] fix: unknown dsh one-shot sidecars steal the raw-key anchor and kill compression (context-explosion root cause) #5687 合入)把「无工具的 dsh 单发请求」改走 verbatim side lane(kernel 不碰);五个既有 dsh 回归 pin 假设 turn 1 走完整管线并在 turn 1 见证 BLOCKED ref,sidecar 化后 turn 1 不再分配 refs → 断言必挂。凡含该 master 提交的 PR head 都继承此红,本 PR 的红 head 恰好含它。
  3. 影响范围与严重程度:纯 CI/测试面,不动发布产物与用户行为;但会把所有基于该 master 的 PR head 打红、卡住合并。对单 PR 是硬阻塞,对外部用户零影响。
  4. 怎么修的(已在分支,非我新增):0470bec62(ci-fixes 顶,前序 agent 按 CI Control Center: Centralized CI Failure Handling & Status Sync (Comment all CI issues here; do not open new issues) | CI 总控: CI 失败集中处理与状态同步(所有 CI 问题在此评论, 勿开新 issue) #4599 协议提交)把这五个 pin 重排到 turn 2(首个完整管线回合)再见证 BLOCKED ref——纯测试改动,无 src/kernel/config/version 变更;随后 e60f2e48 将 ci-fixes 并入本 PR 分支完成继承红收敛。机制成立:turn 1 之前无可折叠内容、无 action surface,把保护点移到首个完整管线回合不削弱对任何折叠的防护。
  5. 是否有回归:无。仅调整测试断言的回合时序(1→2),未触生产代码路径;全矩阵 0 失败佐证无副作用。
  6. 是否破坏缓存:不涉及(纯测试 pin 重排,未触缓存/重试/增量游标/去重键)。
  7. 是否改设计方向:不涉及。
  8. 是否引入新配置:不涉及。

收工判定:无需我再 push——收敛提交已在分支且上游全绿,本 PR 两个 blocker(behind + 红分片)均已清除,可待人工合并(合并权在 owner,我按规不合并)。如需我在 #4599 总控补一行留痕,说一声即可。

@ranxianglei ranxianglei left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(ai: vllm-qwen/qwen3.8-27b)

Formal APPROVE event blocked again by GitHub's own-PR rule (this lane posts under the repo account) — the verdict below stands as the review-lane approval, same convention as round 1.

Scope of this round: facet C commit 6b0d37bae (orphan-journal ghost after authoritative clears) + ci-fixes/master convergence merge e60f2e482 (inherited-red per #4599, master-side changes out of review scope) + review-lane comment fix 2aeac3f20 (comment-only, zero behavior delta vs e60f2e4). Facet A (ref-floor phantom byRef) was fully audited in round 1 and its regression suite re-passes on this head.

① 是什么问题:unified shadow(默认模式)下,凡走过 #2695 archive reap(block 离开 state.blocks、contents 留在 blockContents 供 decompress)的会话,一旦再经历权威清空(Codex/Responses native compaction rebase 原地 clear blockContents),孤儿日志(sessions.orphans_json)里清前条目永久保留 → 每次抽样校验必报 DIGEST DRIFT [diverged: blockContents] → 连续 3 次抽样失败后本进程对该会话的 JSON envelope 写入永久冻结,重启从陈旧快照恢复。

② 什么原因导致:两层。机制层:journal 写入语义是 MERGE(只增不删)——live map 是 journal 内容的唯一事实源,但 store 侧从不删除 live 侧已权威清掉的条目,幽灵跨 clear 存活。触发层:rebase 清正是合法清空,却撞上了只增语义。同 commit 另发现并修复:mysql upsert 此前根本不带 orphans_json 列(列存在但从未写入)→ mysql 会话同一潜在缺陷类、不同机制(journal 恒 NULL,reap shape 在 unified 视图不可表示)。

③ 影响范围与严重程度:走过 archive reap 且随后发生权威清空的会话(Codex/Responses native compaction 路径);shadow 为默认模式,面宽。p1 数据丢失类(与 facet A 同类):冻结期内仅内存保存的回合存活,重启回退到冻结前快照。字段证据 S6(v0.1.212,Codex Desktop)。

④ 怎么修复的:写入语义 MERGE → REPLACE——每次 save 把 live 孤儿集逐字落库(null = 重置),"journal = live set"恒成立。并验证了「thinned map 到不了 plan 构造」这一前提的全链路:LazyBlockContents 任何访问(含迭代)都走 ensure() 从 store 重材化,失败时 throw 而非返回瘦集(src/session.ts:72-74,注释明言"否则一次迭代会把磁盘上的 block contents 抹掉");重材化来源 = block 行 + orphans journal(src/storage/shadow.ts:700-709),所以 journal-only 条目在 #3050 drop 之后也不会被 REPLACE 误删;blockContents 全部变更点逐一审计过——orphan-gc 只删仍留在 state.blocks(inactive)的 block、不涉 journal;convergeToDisk 用磁盘构建的完整视图替换(正是要持久化的内容);rebase clear 就是设计内的重置触发点。reapBlocks 的增量合并保留,与 REPLACE 自洽(其新增项由下一次 save 重新携带)。

⑤ 是否有回归问题:逐项审过既有行为:(a) reap shape 保持可表示(#4216/#4242 不变,更新的 pin 测试已在 PR body 披露);(b) 持久孤儿不被过度清除(负控测试:孤儿跨 21 次 save 保留,两次抽样校验均 clean);(c) 真分歧仍可检出(verifier 比较逻辑未动,digest 机制与 facet A 同一套,facet A 负门钉住 messageRefs 维,blockContents 维同路径);(d) mysql 补上 sqlite 早已带的列——full-mode mysql 用户的 reap-shape contents 此前只活在 JSON envelope、unified 视图缺失(同类漂移/冻结潜在风险),本 commit 关闭;(e) 行为变更旧值→新值:journal 保留策略 merge→replace(已披露)、pin 测试期望翻转(已披露,旧输入形状生产不可达);heal = 升级+重启(per-process frozen set 重置,首个 verify-clean save 恢复 envelope 写),无需 DB 修复。高危模式检查:无 fail-fast 变吞错、无阈值/时序/输出格式漂移。

⑥ 是否破坏缓存:不涉及——未触碰任何缓存/重试/增量游标/去重键;ingest watermark 机制本身恢复正常工作(这正是修复目的)。

⑦ 是否改了设计方向:无——留在既有 unified-shadow 架构内,只是收紧 journal 契约(live set verbatim),数据流不变。

⑧ 是否引入新配置:无(无新 env/config/CLI surface)。

测试有效性:新 tests/issue5589-orphan-journal-ghost.test.ts hermetic red/green(pre-fix 复现字段日志签名 [diverged: blockContents])+ 负控(持久孤儿不误报)——gate 双向断言,#3617 纪律达标;断言有牙(verifyDrift/verifyOk 计数 + 精确日志签名),非恒真。更新 tests/storage-orphan-representable.test.ts step 4 期望并在 PR body 披露。

本地验证(e60f2e4):相邻定向套件 16 个文件 69/69 pass(与 PR body 声明一致)、golden-storage-render 5/5(goldens 在本 head 幂等)、typecheck clean。全量回归以 CI 为准。

kernel:未触碰,无 kernel-change 声明——授权门禁不适用。§7.4 human-tier 已在 PR body 标注。

已顺手修:facet C 把 journal 写入语义改为 replacement 后,src/storage/ingest.ts:435 与 src/storage/shadow.ts:460 两处注释仍写 "the store merges it"(与代码事实不符)——订正为 replacement 语义并引 #5589-C + commit 2aeac3f。

收工判定:可合——新 head 2aeac3f 的 CI 矩阵已绿(26 success / 0 failure / 4 skipped = e2e 门控车道;相对受审树 e60f2e4 仅注释级差异,零行为增量)。请 owner 合入。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Facet C(孤儿日志幽灵)审核完成,结论可合。

已顺手修:facet C 把 journal 写入语义改为 replacement 后,src/storage/ingest.ts:435 与 src/storage/shadow.ts:460 两处既有注释仍写 "the store merges it"(与代码事实不符)——订正为 replacement 语义并引 #5589-C + commit 2aeac3f。

请 owner 合入:#5597

Resolve the stale-ci-fixes tangle left by last turn's convergence merge: adopt
master for every non-#5589 file (the branch's only intended change is the
#5589 orphan-journal / ref-floor storage fix) and keep that contribution
verbatim. Net diff vs master is now exactly the 9 #5589 files.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷

Merge conflict resolved and pushed — verified clean against live sources.

Why it looked bigger than it was: the branch's real change is only 9 files (the #5589 orphan-journal / ref-floor storage fix: src/session.ts, src/state-hydration.ts, src/storage/{ingest,mysql,shadow,store}.ts, + three tests/issue5589-*). But last turn's convergence merged a stale/deep ci-fixes lineage into the branch, so integrating any newer master reopened ~104 conflicting files (including kernel/**) — that's what made the base report dirty.

How I fixed it (mechanical, no design decisions):

Verified live this run:

  • Local npm run typecheck (after installing deps): green — kernel + pi-subagents rebuilt, tsc --noEmit clean.
  • Pushed fast-forward (no force; the head was not concurrently moved). Branch head is now 9a558e7f8e.
  • Live state: mergeable: true, mergeable_state: "blocked" (not dirty), and master c26c31ace is confirmed an ancestor of the branch → no conflict remains. "blocked" is the normal post-push state (branch protection awaiting CI + human approval); agents never merge.
  • CI on the new head: 0 failed, checks still running.

On the repeated "dirty" alerts: they landed after my push and are stale/delayed detections — the live check above shows the conflict is gone. Master did not move past c26c31ace during this run, so nothing re-broke it.


中文八问(结构化结论)

  1. 是什么问题:PR 分支与 base(master)出现合并冲突(状态显示 dirty)。根因不在本 PR——上一轮收敛把一条陈旧/深层的 ci-fixes 谱系合进了分支,使之后并入任何更新的 master 都会重开约 104 个冲突文件(含 kernel/**);本 PR 真实改动只有 9 个文件(Codex Desktop: snapshot-shadow lost-tail and unified shadow state.messageRefs DIGEST DRIFT on 0.1.212 #5589 orphan-journal / ref-floor 存储修复)。
  2. 什么原因导致的:上一轮「Merge branch 'ci-fixes'」把哨兵车道旧内容拉进分支,形成纠缠拓扑,分支有效基线变深;于是与最新 master 的三方合并在大量文件上双方都改动而冲突。非本 PR 代码错误。
  3. 影响范围与严重程度:仅影响该 PR 可合并状态(卡住、CI 车道阻塞),不影响已发布版本或线上行为;属「PR 卡住」级,非用户可用性缺陷。
  4. 怎么修复的:机械消解——先确认所有 Codex Desktop: snapshot-shadow lost-tail and unified shadow state.messageRefs DIGEST DRIFT on 0.1.212 #5589 改动都在自动合并区(唯一例外是 src/state-hydration.ts 的 REF_FLOOR_RAW_ID,需真并集);对约 99 个非 Codex Desktop: snapshot-shadow lost-tail and unified shadow state.messageRefs DIGEST DRIFT on 0.1.212 #5589 文件直接采用 master 原样(净 kernel 差异为零,kernel-guard 不触发);对 5 个贡献源文件逐块按「master 更新」取值+该并集。最终得到一个双亲合并提交,相对 master 的 diff 恰为那 9 个 Codex Desktop: snapshot-shadow lost-tail and unified shadow state.messageRefs DIGEST DRIFT on 0.1.212 #5589 文件。
  5. 是否有回归问题:无。除并集外全取 master 原样(等价于 master 已验证行为),并集只保留两边各自独立存在的定义;本地 typecheck 全绿;未改任何阈值/默认值/时序/输出格式/wire 形状。
  6. 是否破坏缓存:不涉及。纯合并冲突消解,未触碰缓存/重试/增量游标/去重键逻辑。
  7. 是否改了设计方向:不涉及。无架构/边界/数据流变化,只是把分支重新对齐最新 master 并保留 Codex Desktop: snapshot-shadow lost-tail and unified shadow state.messageRefs DIGEST DRIFT on 0.1.212 #5589 修复。
  8. 是否引入了新的配置:不涉及。无新 env/config/默认值。

判定:合并冲突已解决并推送(分支头 9a558e7f8e),本地 typecheck 绿,线上核实 mergeable=true、非 dirty。剩余仅 CI 跑完 + owner 人工批准合并(agent 不合并)。若之后 master 再前进且触及这 9 个文件,可能需再收敛一次(属哨兵/ci-fixes 车道职责)。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] (ai: vllm-qwen/qwen3.8-27b)

重启说明与当前状态确认:review-worker 进程曾卡住 31 分钟被强制重启。卡住前我的审核工作已全部落定(结论见上楼层 12/14/19:主修复 APPROVE、facet C 可合、注释级顺手修 commit 81b36c87c),无未交付的阶段性成果丢失;卡住期间没有产生需要补做的动作。重启后我重新核实了终态(全部为刚才的实时读取,非引用旧数据):

  • 分支头:9a558e7f8e(ci-engineer 消解冲突后的双亲合并提交,相对 master 的 diff 恰为 9 个 Codex Desktop: snapshot-shadow lost-tail and unified shadow state.messageRefs DIGEST DRIFT on 0.1.212 #5589 文件)。
  • CI:该 head 的 check-runs 共 30 项 —— 16 success / 14 skipped(e2e real-client 门控车道,默认跳过)/ 0 failed / 0 pending,全绿。
  • 冲突状态:上游 master 仍在 c26c31ace4,已是分支头的祖先(即分支已包含最新 master),不存在新的冲突源;此前反复出现的 dirty 告警是陈旧/延迟检测,实况已干净。

收工判定:本 PR 两个 blocker(behind + 红分片)均已清除,CI 全绿,可合并。按规 agent 不执行合并——请 owner 手动合入:#5597 。若合入后 master 再前进且触及这 9 个文件才需要再收敛(属哨兵/ci-fixes 车道职责),目前无需任何进一步操作。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷

Re-verified live this run — the fix is unchanged and CI has since finished:

  • Branch head still 9a558e7f8e; master still c26c31ace (did not advance), so master remains an ancestor of the branch → no merge conflict.
  • mergeable_state: "blocked" but CI is now complete: 30/30 checks, 0 failed, 0 pending (green). "blocked" here is the branch-protection human-approval gate — neither a conflict nor a CI failure.
  • The repeated "dirty" alerts were stale/delayed detections of the pre-push state; there's nothing left for the CI lane to fix.

Remaining step is owner approval + merge (agents don't merge). I won't re-post on every duplicate trigger unless the state actually changes (a new conflict or a red check).

中文:冲突已解、CI 全绿(30/30,0 失败),现仅剩 owner 人工批准合并;重复的 dirty 告警是推送前旧状态的延迟检测,无新可修项。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷

Closing this CI lane — final state (verified live):

  • Merge conflict: resolved & pushed (head 9a558e7f8e); master c26c31ace is an ancestor → mergeable: true, no conflict.
  • typecheck: green locally (kernel + pi-subagents rebuilt, tsc --noEmit clean).
  • CI: 30/30 checks passed, 0 failed, 0 pending.
  • Only remaining gate: human approval + merge (mergeable_state: "blocked" = branch protection). Agents don't merge.

Nothing further to fix on my side — standing down unless the state regresses (a new conflict or a red check).

中文:冲突已解并推送、本地 typecheck 绿、CI 30/30 全过,现仅剩 owner 人工批准合并;本车道收工,若状态回退(新冲突或红 check)再介入。

ci-check added 2 commits November 17, 2026 23:27
…elds (#5589)

The sampled DIGEST DRIFT diagnosis (#5677) names the divergent top-level
fields of the first divergent lastMessages slot, but a fixed-size
serialization difference INSIDE a sidecar field (e.g. rawResponsesItem —
the v0.1.215 Codex CLI shape: same id/role/text, +26 canonical bytes in
rawResponsesItem) still leaves the exact wire member unknown.

Drill one level: for each divergent top-level field whose value is a plain
object on both legs, name the inner members whose canonicalized values
disagree. Wire schema names only — values never reach the log line (same
privacy contract as the #5677 field names). Bounded: at most 5 inner
members per field, sorted. Fields absent on one leg are not drilled (the
top-level name already says "the sidecar appeared/disappeared"); arrays
are not drilled (index-level diffs are a different shape).

The line prefix stays byte-identical for existing greps — the nested
segment is additive:
  ...; fields: rawResponsesItem; rawResponsesItem: summary

Wired into both call sites (the header-level describeViewDivergence and
the per-row compareMessages that produces the sampled verify line).
Diagnostic only — no behavior, wire, storage, or config change.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working codex p1 scope/storage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Codex Desktop: snapshot-shadow lost-tail and unified shadow state.messageRefs DIGEST DRIFT on 0.1.212

2 participants