Skip to content

[PR] storage: shadow DIGEST DRIFT now names the diverged dimension and slot; prefix rewrites fail soft at the delta boundary - #3948

Merged
ranxianglei merged 2 commits into
masterfrom
2026-10-20_issue3919-shadow-drift-forensics
Oct 20, 2026
Merged

ranxianglei merged 2 commits into
masterfrom
2026-10-20_issue3919-shadow-drift-forensics

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Related: #3919

Background

In #3919 the shadow-mode sampled-verify backstop fired 28 times across four Codex Desktop sessions on v0.1.201. Its log line said "views differ" but not WHERE they differ, so real interior-history rewrite vs comparison asymmetry vs non-convergence could not be told apart; v0.1.201's verifyOk/verifyDrift counters are also in-memory-only, so even consecutive-failure detection was impossible from logs. This PR adds the forensics and closes one concrete drift window.

Changes

  1. Drift forensics (src/storage/shadow.ts): on detected drift the log now appends which top-level view dimensions diverge and, for lastMessages, the first divergent slot with both sides' id/role/serialized-length/sha8 fingerprints — never body text. A session still drifting on the NEXT sample is flagged (repeat: consecutive sampled drift — the views are not converging). Diagnostics run in a try/catch so they can never break the heal path.
  2. Head-slot probe (src/storage/store.ts + src/storage/ingest.ts): planIngestDelta takes an optional head-slot identity; when trustedLen > 1 the delta path reads the stored msg_id of message seq 0 (PK point read, zero serialization cost) and refuses the delta if it changed, failing soft to the full path on the same save. Previously such a rewrite rode the delta until the next sampled verify (window <= 20 saves). Same-id byte drift below the boundary stays the sampled backstop's job (documented trade-off).

Behavior changes (disclosed)

  • Log format: the [storage] unified shadow DIGEST DRIFT ... line gains an additive forensic suffix (e.g. [diverged: lastMessages; tail slot 0: expect h_msg0/user len=67 sha=e24a6f57, got tampered/user len=70 sha=c18830b2]) plus the optional (repeat: ...) marker. The prefix stays byte-identical, so existing greps keep matching.
  • Delta path: a save whose first tail slot below the boundary changed identity now refuses the delta and takes the full path on the same save (convergence window for prefix-rewrite shapes: <= 20 saves -> 0). Churning workloads may see slightly more full ingests. No wire / persistence / config / watermark-semantics change.

Tests

New tests/issue3919-shadow-drift-forensics.test.ts (3 tests): head probe end-to-end + planner level + negative control (metadata-only raw-tail saves still delta); CAS-tamper scenario -> diagnosis names exactly lastMessages / slot 0 with differing fingerprints and no body text in any output; repeat-marker lifecycle (flagged while consecutive samples drift, cleared after a clean sample). Pre-existing tests/storage-ingest-watermark.test.ts (16 tests incl. the original CAS-tamper scenario) passes unchanged — the head probe does not interfere with it (its row columns stay intact).

No config surface change. No kernel change. No version bump.

@ranxianglei ranxianglei changed the title storage: shadow DIGEST DRIFT now names the diverged dimension and slot; prefix rewrites fail soft at the delta boundary [PR] storage: shadow DIGEST DRIFT now names the diverged dimension and slot; prefix rewrites fail soft at the delta boundary Oct 20, 2026
@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-20_issue3919-shadow-drift-forensics (52961ef)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.4750-pr.3948.a37159c

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr3948.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复 —— 本 PR 是 #3919 的修复交付物(Related: #3919),与 #3919 线程里声明的两项工作(对账点取证 + delta 路径廉价前缀探针)一一对应;按失败签名(DIGEST DRIFT 日志行、delta head probe、issue3919-shadow-drift-forensics 测试名)检索 open+closed issue/PR 无其他在途条目。同家族条目 #3407/#3285/#3288/#3559/#3841 已在 #3919 triage 中逐一排除(机制不同)。

标签: bug / codex / scope/storage / p2(修复 #3919 缺陷;受影响客户端为 Codex Desktop;统一存储对账路径;严重程度随 #3919——shadow 模式自愈合、不影响会话可用性,代价是 CPU 与日志噪音)。

审核进行中:分支与 master 同步(落后 0 commit),diff 为 4 文件 +419/-6(src/storage/{shadow,store,ingest}.ts + 新测试)。已核对 fail-soft 语义、位置假设、CAS 持久化机制;定向测试与 CI 状态确认后补完整八问结论。

@ranxianglei ranxianglei added bug Something isn't working codex p2 scope/storage labels Oct 20, 2026
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Review 完成(head 0dd6f221f,含 commit 2 的 unused-exports 修复)。结论:代码质量达标、方向正确、未发现回归;CI 唯一红是一个与本 PR 无关的 dsh-native 超时 flake(本地同 head 复现通过),建议 re-run 后合入。

查重:不重复(首条已述)。本 PR 是 #3919 floor 6 承诺的两项交付物本身,无并列在途项。

结构化八问

  1. 是什么问题:shadow 模式的 DIGEST DRIFT 日志只说「views differ」不说哪里不同——真实的历史内部改写、比较不对称、还是持续不收敛,从日志上分不开;同时前缀改写类漂移要搭 delta 路径一路骑到下一次采样校验才被发现(窗口 ≤20 次保存)。
  2. 什么原因导致的:两层。(a) drift 检测当初只设计了计数器,日志行没带取证信息;(b) delta planner 按设计只点验边界槽(trusted 尾行,成本考虑),头部槽的身份无人验证——这是 delta 路径的已知设计边界,不是实现错误。
  3. 影响范围与严重程度:p2。不影响可用性——backstop 本身工作正常(watermark 清除 → 全量重灌自愈)。受影响面:shadow 模式会话的可诊断性 + 前缀改写形态的收敛窗口(≤20 saves)。修后:取证立即可用,前缀改写收敛窗口 ≤20 → 0。
  4. 怎么修复的:两个机制。① 取证:drift 检出时对 unified replay 视图与 legacy load 视图做逐顶层维度 canonicalized-JSON 等值比较,点名分歧维度 + lastMessages 首个分歧槽位,两侧各带 id/role/serialized-length/sha8 指纹(永不带正文文本);同一会话下一采样仍漂标记 (repeat:);诊断整体 try/catch,永不破坏 heal 路径。② 头部探针:planIngestDelta 新增可选 head-slot 身份参数,trustedLen > 1 时以 PK 点读(seq 0 行的 msg_id,零序列化成本)取存储身份,变了就拒绝 delta、同一次 save 内 fail soft 走全量路径。
  5. 是否有回归问题:逐项审过,无意回归(清单见下)。
  6. 是否破坏缓存:不涉及模型侧缓存/前缀缓存。watermark 语义不变(probe/clear/re-establish 路径未动),delta 路径只多一个拒绝条件(已披露);CAS 去重/refcount 未动。
  7. 是否改了设计方向:没有——仍是双写 + 采样 backstop 架构,只是加了观测面和一个 delta 门禁。
  8. 是否引入了新的配置:没有。PR 声明「No config surface change」,我核对属实:无新配置字段/env/CLI flag,config-surface-guard gate 绿。

回归审计(触碰路径的既有行为清点)

既有行为 判定
Digest 契约(expectView/gotView 的计算) 保持。仅变量改名,计算逻辑逐行等价;golden corpus 套件(corpus round-trip digests equal the legacy manifest,node:sqlite 变体)通过
Watermark 语义 保持。probeIngestBoundary/clearIngestWatermark/ingestWatermark 未动;既有 14 个 watermark 测试原样通过(boundary tamper 回退、fold 容忍、foreign rows、rewound tail、snapshot 会话走全量等全部覆盖)
DIGEST DRIFT 日志格式 有意变更(已披露):前缀到 unified replay != legacy load view 为止字节级不变(我核对了插入点),现有 grep 不受影响;追加 forensic 后缀 + 可选 (repeat:)
Heal 路径顺序 保持。verifyDrift++ → repeat 判定 → add → lastError → clearIngestWatermark → 诊断(try/catch)→ 日志,与 master 相对顺序一致;诊断即使抛异常也不影响 heal
Delta 路径 有意变更(已披露):多一个拒绝条件(head 身份变化或行缺失 → 全量路径)。稳态 append、metadata-only raw-tail save 仍走 delta——负控测试钉住了这一点
导出面 commit 2 移除两个内部 interface 的 export(函数保留供测试),unused-exports gate 绿

高危模式逐项过:无 fail-fast 变吞错(try/catch 只包诊断格式化,不包 heal 逻辑);无阈值/时序漂移(VERIFY_EVERY_INGESTS=20 未动);输出格式变化仅限上述已披露的日志后缀。无 wire/持久化/版本变更——与 PR 声明一致,属实。

测试有效性

三个新测试都有牙,非摆设:

  • head probe:端到端(真实 store + UnifiedShadow.record)+ planner 级正/负/无-head 三向控制;断言 delta 计数、view deepEqual 收敛、watermark 长度——既抓过度拒绝(稳态 append 必须仍走 delta)也抓漏拒(重写头部必须落全量)。
  • CAS-tamper 取证:断言点名恰好 lastMessages/slot 0、两侧指纹不同、且 !line.includes(正文) 且不含原始 hash——「永不带正文」这条安全性质被显式钉住。
  • repeat 生命周期:三阶段(drift/repeat/repair)+ flag 成员关系 + 精确 marker 字符串,清理后不再出现。

全部 hermetic/tmpdir/env 钉死,确定性成立。本地定向跑(5 个相关文件):37 tests,36 pass,0 fail,1 skip(skip 为 better-sqlite3 可选原生依赖本机未装,环境性,node:sqlite 变体通过)。typecheck 干净。

CI 状态与唯一的红

head 0dd6f221f:全部静态 gate 绿(static-gates ×2 OS/node、unused-exports、build-artifact、docs-sync/coverage、version-guard、lock-sync、kernel-guard、lane-guard、pi-subagents-guard、config-surface-guard、test-import-graph、plan),test shard 5/6 绿。唯一红:test (windows-latest, 22, 2),失败项为 #2604: a failed title POST re-arms the dedup so the next event retries(tests/dsh-native.test.ts:2639),错误 timeout waiting for retry POST——waitFor 15s 超时,不是断言失败。我的判定:与本 PR 无关的 runner 负载 flake,证据链:

  1. diff 只碰 src/storage/{shadow,store,ingest}.ts + 一个新测试文件,dsh 标题推送路径(/__bili/plugin/session-name 及其 dedup)完全不在触碰面内;
  2. 同一 run 里同文件相邻测试(fix: stop truncating Windows launches at spaced paths (#679) #682/[bug] compress 保护区过滤在完整性调整之后挖洞,导致 assistant tool-call 消息缺 reasoning_content(DeepSeek 思考模式 400) #684/fix(#638): codex plugin install self-heals malformed blocks instead of short-circuiting on the origin line #685/E2E verify on real Windows: codex accepts CODEX_HOME overlay and registers the bili MCP server (#681 / PR #683) #686,含同一 dedup 机制的其他侧面)全部通过;
  3. 我在该 head 本地单测复现,71ms 通过;
  4. 仓库无此失败签名的存量 issue;分支落后 master 0 提交。

处置建议:re-run 该 job 即可;若再次复发则是另一个症状/机制,届时单独处理,不应绑在本 PR 上。

两个非阻塞发现(记录在案,不拦合并)

  1. repeat marker 措辞边界:一个采样窗口(≤20 saves)内发生两次独立的一次性 churn(第一次已被全量重灌治愈、中间没有干净采样),第二次会被标成 (repeat: ... the views are not converging)——事实部分「consecutive sampled drift」成立,但「not converging」这一解读在该场景下略微过强。缓解:forensic 后缀对每次事件都点名维度/槽位/指纹,对比两行日志即可分辨。该字符串是 PR 正文声明的验收标准的一部分,我不单方面改——留给作者/owner 决定是否软化措辞(如改成「no clean sample in between」)。
  2. 已知残余竞态(文档化边界,非缺陷):竞争写入者在事务外探针与 BEGIN IMMEDIATE 之间重写 head 行的话,仍会搭 delta——事务内再验只覆盖边界槽。这与现有 pre-txn 探针的设计哲学同类,由采样 backstop 兜底,PR 正文已把这类 trade-off 写明。可接受。

代码整洁硬标准逐项过:命名达意、无死代码、无重复、错误处理带上下文、注释讲为什么、风格与文件既有代码一致、无新依赖、函数规模合理——全过。

收工判定

可合入:方向正确(#3919 验收标准逐项兑现)、无回归、测试有牙、行为变更全部披露且我逐一核实。待办:re-run test (windows-latest, 22, 2) 确认 flake 转绿(我的判断是不用改码)。合并按规矩留给人工。#3919 保持 open 等下一轮运行时数据,符合 floor 6 的安排。

awork added 2 commits October 20, 2026 21:04
…x rewrites at the delta boundary (#3919)

The sampled verify backstop fired 28 times across four Codex Desktop
sessions in v0.1.201 but only said 'views differ' — nothing about WHERE
they differ, so interior-history mutation vs comparison asymmetry vs
non-convergence could not be told apart, and v0.1.201's in-memory-only
counters made even consecutive-failure detection impossible from logs.

- shadow.ts: on detected drift, log which top-level view dimensions
  diverge and, for lastMessages, the first divergent slot with both
  sides' id/role/serialized-length/sha8 fingerprint (never body text);
  a session still drifting on the NEXT sample is flagged '(repeat: ...)'
  so non-convergence is visible in the log itself. Heal path, digest
  contract, and counters unchanged.
- ingest.ts + store.ts: planIngestDelta takes an optional head-slot
  identity; when trustedLen > 1 the delta path probes message seq 0's
  stored msg_id (PK point read) and refuses the delta if it changed,
  failing soft to the full path on the same save instead of riding the
  delta until the next sampled verify. Same-id byte drift below the
  boundary stays the backstop's job.
…exports gate)

DriftSlotFingerprint/DriftDiagnostics are referenced only inside
shadow.ts — consumers use the functions structurally. Drop the leading
export per the gate's own fix guidance instead of waiving.
@ranxianglei

Copy link
Copy Markdown
Owner Author

Rebased onto current master (step-3 had renamed the store calls to the sync fast lane inside record() — the drift-diagnosis block now rides the new structure; this.store → this.sync at the head-boundary probe). Verification: tsc 0, issue3919 forensics 3/3, storage-unified 14/14, full-flip 8/8, reap-journal 2/2, archive-gc 5/5, media-cas 3/3, backend-contract 6/6.

@ranxianglei
ranxianglei force-pushed the 2026-10-20_issue3919-shadow-drift-forensics branch from 0dd6f22 to 52961ef Compare October 20, 2026 13:04
@ranxianglei
ranxianglei merged commit 74090ac into master Oct 20, 2026
29 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.680-master.74090ac (rolling master tag).

npm install -g billion-context@0.0.680-master.74090ac  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working codex p2 scope/storage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant