Skip to content

[PR] fix(persistence): publish concurrent snapshots through immutable journals - #5480

Open
Maziar123 wants to merge 10 commits into
ranxianglei:masterfrom
Maziar123:fix/concurrent-snapshot-persistence
Open

Maziar123 wants to merge 10 commits into
ranxianglei:masterfrom
Maziar123:fix/concurrent-snapshot-persistence

Conversation

@Maziar123

@Maziar123 Maziar123 commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Affinity/order snapshots used a shared temporary filename and process-local read/merge/write coordination. Unique temporary filenames alone would still permit lost updates from two stale views.

Change

Node20-standard-library immutable publications: fsync payload -> fsync private staging directory -> atomic rename to a unique publication directory -> fsync journal parent -> reclaim only exact valid consumed inputs -> fsync parent. Readers union valid leaves with the unchanged legacy baseline using the existing merge policy. Corrupt/unreadable inputs and crash-left staging directories are retained, not age/PID-reaped or recursively removed. Failed/unsupported directory barriers never authorize reclaiming inputs.

Verification

Candidatefb0ccf72:33 focused journal/affinity/restart runtime checks passed, plus TypeScript, unused-export and test-import-graph gates. Separate stale-view publication smoke preserved both independent updates through compaction and reopen. Coverage includes real concurrent writers and injected crash/barrier failures, not a power-loss certification.

Limits and migration

No SQLite/native dependency or kernel/wire change. The payload format remains compatible, but the disk layout changes: backups must include the baseline AND its journal. Older readers do not see post-journal updates. An unsupported barrier may leave a visible publication without proven power-loss durability. Corrupt inputs and abandoned staging debris are not globally disk-bounded. No Windows/network-filesystem runtime result is claimed.

Related work

#1737 fixed a sequential stale-merge case; it does not serialize simultaneous processes. This is the concurrent-publication follow-up, related to#1724/#3397, not a claim to explain every historical restart incident.

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重:不重复。 本 PR 是已关闭的 #1737(顺序 stale-merge)/ #1724 / #3397 的并发发布后续——机制不同:那一批修的是「单进程顺序读→改→写」,这里修的是「两个进程同时对同一个共享 .tmp 做读改写、后写的 rename 覆盖掉先写的合并结果」。检索了 open+closed 的 issue 与 PR,没有同机制(不可变发布日志)的在途或已立项条目;引用的三个 issue 均已关闭,本 PR 是在其之上的跟进而非重复。

初步判断(完整结构化八问结论随后补): 方向正确、实现质量高。核心机制把 affinity/order 两个快照从「单一 <name>.json.tmp rename 覆盖」改成追加式不可变发布:每次写都生成一个永不复用的 pub-* 叶子目录(私有 staging → fsync → 原子 rename → fsync 父目录),读者对「冻结基线 + 所有存活叶子」取并集,且只回收那些确已解析并全部并入新输出的旧输入。这从根上消除了丢失更新——再没有任何一次写会整体覆盖别人已发布的合并结果。

接下来按规范逐项核:①机制正确性(回收守恒 / 崩溃边界 / 目录屏障失败)②相对既有行为的回归(重点是磁盘布局变更)③外部提交质量门槛 ④测试有效性(并发与崩溃注入是否真有断言牙齿)⑤CI。

已先行确认一点:生产代码里除 affinity-persist.ts / order-persist.ts 之外没有任何地方直接读 prefix-affinity.json / session-order.json,所以「基线文件从此冻结、不再被改写」不会波及其它读取路径;引用这两个文件名的测试本 PR 也已同步更新。CI 目前镜像尚未缓存到结果,我会查一次。

@Maziar123

Maziar123 commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor Author

Follow-up at b0ddbd07 addresses the Windows review findings without claiming durability that the filesystem cannot provide:

  • A failed directory-durability barrier reclaims nothing.
  • uncovered reports the entire protected census on that failure; retained counts only census leaves that still exist.
  • The reused-journal regression now uses the correct three-leaf census.

Verification: 25/25 focused tests, typecheck, file-size, and diff checks passed. Filesystem capability is tested explicitly; unsupported barriers do not produce a power-loss durability claim.

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

已对跟进提交 b0ddbd07 做了完整复审(本地只读核对 + 定向跑测试,未改动/未推送任何分支——分支归你)。结论:Windows 侧的三点意见都落实了,而且顺带修掉了我上一轮发现的一个潜在丢数据路径。质量达标,建议合入(等 CI 绿)。

查重(重申):不重复。 本 PR 是已关闭的 #1737(顺序 stale-merge)/ #1724 / #3397 的并发发布后续,机制不同(不可变发布日志 vs 顺序读改写),open+closed 无同机制条目。


结构化八问

1. 是什么问题
两个共享同一 state dir 的实例同时改写 prefix-affinity.json / session-order.json:都走「单一 .tmp 读→改→rename」,进程内 writing 标志挡不住跨进程,后 rename 的一方把先写方已合并的内容整份覆盖掉 → 丢失更新(#4125/#1724/#3397)。

2. 什么原因导致的
并发下对同一个文件名做整体 read-modify-write,没有任何跨进程的不可变发布边界;rename 是唯一的原子点,但它覆盖的是别人的合并结果。

3. 影响范围与严重程度
多实例(dsh profile / 多 launcher)共用默认 state dir 时触发;单实例不触发。严重度:p1 级可用性缺陷——会话亲和链 / persona 骨架在重启+并发后静默丢失,匿名会话无法再 reattach。属「能用但会丢状态」。

4. 怎么修复的
改成追加式不可变发布:每次写生成一个永不复用的 pub-* 叶子目录(私有 staging → fsync payload → 原子 rename → fsync 父目录),读者对「冻结基线 + 所有存活叶子」取并集;回收前用 store 自己的 covers() 判定规则逐叶校验该输出的记录是否真的覆盖了输入,只回收被覆盖且字节仍一致的叶子(stillParsed 做 TOCTOU 兜底);损坏/不可读输入一律保留不删。跟进两笔(cc07651 + b0ddbd0)补上了关键一环:有界(LRU 截断)输出不再删除它没覆盖到的叶子,屏障失败时如实上报 uncovered(在持久性分支之前算好,不会谎报 0)、reclaimed 恒为 0、retained 只数仍在盘上的叶子。

5. 是否有回归问题
审过的既有行为逐项过:无有害回归。合并规则(deeper/fresher)、LRU 上限 1024、order 字节预算、500ms/5s 防抖全部不变;hydrate 对旧单文件/新安装向后兼容;仓库里除这两个模块外没有任何生产代码直接读这两个 json,所以「基线冻结不再改写」不波及其它路径;不涉及 wire / 消息结构(plugin/proxy 两种压缩模式均不受影响);本 PR 不动 kernel / 不改配置。⚠️ 需明确的行为变更(旧→新,已写入 CHANGELOG):①磁盘布局从单文件变为「冻结基线 + .journal/pub-*/snapshot.json 活叶子」,备份必须两者都带;只删 .journal 会退回最后基线(一致、非损坏);旧版本读者看不到 journal 里的新写入。②损坏文件自愈被移除——旧代码下次 flush 会用有效内容覆盖坏文件,新代码永久保留坏文件字节并把新数据走 journal(更安全但不再自愈)。③(轻微)迁移时冻结进基线的记录会逃过 LRU 淘汰、每次 hydrate 复活(旧单文件是永久丢弃),受上限约束、无害偏有益。④(新增)有界写入不再删掉它没覆盖的叶子 → 持续 churn 下叶子可能多留一会儿才回收,仍受上限约束。

6. 是否破坏缓存
不涉及。这是宿主侧持久化,与模型前缀缓存无关。

7. 是否改了设计方向
架构未转向:仍是「机器全局单一 state dir + 每 lane 单写者」契约,只是把两个快照的落盘从「整份覆盖」换成「不可变追加 + 覆盖校验回收」。新增的 covers 让回收判定与 store 的支配规则绑定,是既有不变量的自然延伸。

8. 是否引入了新的配置
无新增 env / config / CLI flag。journal 始终开启、无开关(可接受);未改任何现有字段语义。


CI 与验证

  • 旧 head(fb0ccf7)全红是假信号:11 个 test shard 全部在 pretest 门禁处 abort,一条测试都没跑,原因仅是新测试 tests/snapshot-journal.test.ts spawn 子进程却没登记进 scripts/live-files.mjs 的 LIVE_FILES(SPAWN_RE tripwire)。非继承 master 红、非代码缺陷。你已在 b0ddbd0 登记(line 119)——我已复现该门禁逻辑确认现在 unregistered spawners = []。
  • 本地定向验证(b0ddbd0 检出,装依赖 + 重建 kernel dist):三个相关套件 snapshot-journal + affinity-persist + issue3397-persona-restart-stability 共 40/40 通过(你说的 25/25 就是 snapshot-journal 单文件;加上另两套是 40)。含真实 4 进程 IPC 读屏障并发、崩溃态构造、EPERM/EIO 屏障注入、有界输出保留、逐记录支配、两段式回收、真实 affinity store 集成——断言都有牙。
  • 分支已并入最新 master,不再是 behind。

两点非阻塞观察

  • coverIndexFor/coverIndex 模块级可变缓存:正确(同步循环 + 每次写新数组引用),略非常规但有注释说明,OK。
  • Windows/NFS 无可用目录屏障时无法证明持久性 → 永不回收 → 叶子增长;PR 正文与 architecture.md 已如实声明「非全局磁盘有界」,属接受的取舍。

判定

外部提交质量门槛(⑤b)通过:命名清晰、无死代码、公共逻辑抽到 snapshot-journal.ts、错误处理完整不吞错、注释风格与仓库一致;跟进两笔正确回应了评审意见并额外堵住了一个真实的 LRU 截断丢数据路径,比第一版更稳。不具备 auto-merge 资格(持久化布局变更 + 新增模块)→ 需 owner 人工合入。我的建议:批准,待 CI 转绿即可合入。

Maziar123 and others added 2 commits November 6, 2026 13:22
…ies already present in master

The branch's original changelog edit copied the two most recent master
entries (ranxianglei#5412 async-storage self-heal, ranxianglei#5415 absorb-debt nudge); after the
branch re-merged master they existed twice in CHANGELOG.md. Merging would
have shipped duplicate release-notes entries for changes that are already
in master (code included). Removes the two added copies plus the separating
blank line; no other change.

@ranxianglei ranxianglei left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(ai: vllm-qwen/qwen3.8-27b)

APPROVE — 独立审完全部 9 个文件(+1850/−193,base be611a856 → head a3d6a94)。机制正确、测试有牙、行为变更已披露。一处小问题已顺手修掉(见「已顺手修」)。

结构化八问

  1. 是什么问题:两个持久化快照(prefix-affinity / session-order)在同一 state 目录被多实例并发写时,共享单个 <name>.json.tmp + 进程本地读写协调,后 rename 者用自己的合并结果整文件覆盖先写者的结果——丢更新(#1724 家族;#1737 只修了顺序场景的 stale-merge,没解决并发 RMW 本身)。
  2. 什么原因导致的:可变单文件 + 整文件替换的发布模型没有跨进程互斥;即使 tmp 名唯一,两个陈旧视图各自 rename 仍互相覆盖。根因是发布模型,不是锁不够多。
  3. 影响范围与严重程度:仅多实例共享同一 state 目录的部署(单实例无感知);严重度 = 特定条件下丢亲和链/会话顺序(前缀缓存退化、排序骨架丢失),不崩溃不阻断——本 PR 是该类 p2 缺陷的根治。
  4. 怎么修复的:换成不可变发布日志——私有 staging 目录写满并 fsync → 原子 rename 到唯一 pub-* 目录 → fsync 父目录 → 仅当「新输出逐键支配且旧叶经字节复核仍在盘上」才回收该输入;读取端取所有存活叶与冻结基线的并集。丢更新在结构上不可能:每轮输出都是新叶,旧叶只在被证明覆盖后消失。
  5. 是否有回归问题(逐项清点既有行为):合并规则/深度优先/LRU 1024/8MiB 字节上限/500ms 去抖/5s 最大延迟——逐行核对未变;hydrate 并集语义保持;wire/配置面/env/依赖/kernel/ /update.ts/version 均未触碰。fail-fast 未变差(parse 从「过滤坏条目导入其余」改为「整份抛错」,更严);无吞错(不可读输入保留+有界告警);日志措辞变化见下方披露。有意行为变更逐条列于披露节。
  6. 是否破坏缓存:不破坏。亲和链内容语义不变(同一会话同一链),只是落盘位置变化;多实例下反而消除了互相覆盖引起的前缀缓存抖动。
  7. 是否改了设计方向:是,局部且合理——状态文件从「可变电文」改为「不可变日志 + 冻结基线」,与仓库账本式身份哲学一致;纯持久化层内部,未跨架构边界。
  8. 是否引入新配置:无新 env/config/CLI。磁盘布局变化(备份需含 .journal/)已在 PR 正文与 CHANGELOG 披露。

行为变更披露(旧值→新值)

  • 落盘布局:<name>.json 每轮重写 → 升级时刻冻结为基线、永不改写;新状态进 <name>.json.journal/pub-*/。全新安装从此不再创建该 JSON 文件。备份/清理必须连同 .journal/(只删 .journal/ = 回退基线,不是损坏)。旧版本读者看不到 journal 写入(PR 已披露)。
  • 损坏文件处理:旧 = 过滤坏条目导入其余 + 下次 flush 重写自愈;新 = 整份永久保留原样、贡献零记录、每次 census 一条有界告警(incomplete 粘滞)。刻意的「守恒优先于自愈」取舍,模块文档与测试均已钉死。
  • 无断电安全文件系统(Windows/部分网络挂载):每次发布走非耐久分支,回收恒为 0 → pub-* 目录随写累积、永不删除(旧实现为单个有界 ≤8MiB 文件)。作者已披露;见观察项 O1。
  • 等值 tie-break 方向:旧 mergeWithDisk 平局本进程胜 → 新 mergeEntries 平局磁盘侧胜。实际不可观测(平局即同记录;变异必带严格更新的 lastSeen)。

观察项(非阻塞,不建议在本 PR 处理)

  • O1 非耐久文件系统上的无界累积:频率受去抖限制(持续流量 ≤ ~1 目录/5s),但长期运行会涨盘;建议后续单独跟踪(配额/周期压实策略),不扩大本 PR 范围。
  • O2 sticky incomplete + 每轮告警:一个损坏叶或一次半删崩溃会使 incomplete 粘滞、此后每次写都打告警——有界但持续的排障噪音,同样建议后续单独跟踪。
  • O3 Record<string, any> 子进程裁决类型:仓库内已有 ≥5 处同类先例(tests/fix-806-malformed-body.test.ts 等),放行。

测试有效性

真多进程并发(4 writer 经 IPC 共同读屏障,逐 writer 断言 consumed===1——核心丢更新回归的钉子);构造态崩溃注入(rename 前/后/半删,不依赖真实 kill 时机);EPERM/EIO 屏障故障注入(reclaimed=0、uncovered 如实上报、旧输入一字不动);能力探测驱动双向断言(dirFsyncWorks() 按实际 FS 能力而非平台名分叉——两种平台都不跳过也不豁免删除);真实 store 集成(1024 上限越界的叶必须留存可读、后续更宽 census 仍可见、真正覆盖后才回收)。issue3397 用例从「flush 自愈」翻转为「基线不可变 + journal 承载」,守恒语义钉死。判定:真测试,断言有牙。

已顺手修

已顺手修:CHANGELOG.md 重复条目 #5412/#5415(两行在 master CHANGELOG.md:16-17 已存在、代码也已在 master;分支早期基于旧 master 写入副本,两次 merge 后成双份——合并会让发布说明重复)commit a3d6a94(3 行,无其他改动)

CI

审查时镜像缓存显示原 head 83c097f 的检查运行中(0 failed / 19 queued / 2 skipped),车道判定绿。本修复 commit 触发新一轮 CI,合入以新一轮为准。

结论:可合(APPROVE) — 新 head CI 绿后即可直接合并。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

review 完成:APPROVE 已提交(新 head a3d6a94,review 5799891897);已顺手修:CHANGELOG.md 重复条目 #5412/#5415(master 已有、代码也已在 master,分支两次 merge 后成双份)+ commit a3d6a94(3 行);无 kernel/ 触碰、无需决策;行为变更(落盘布局/损坏处理/非耐久 FS 累积/tie 方向)已在 review 中逐条披露,两项非阻塞观察(O1 Windows 无界累积、O2 sticky incomplete 噪音)建议后续单独跟踪。CI 在新 head 重跑中,绿后即可合。

@ranxianglei ranxianglei left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(ai: vllm-qwen/qwen3.8-27b)

APPROVE(新 head a3d6a94 复审)

自上一轮批准以来分支上只有一个提交:a3d6a94(本审查车道的顺手修复——删除 CHANGELOG.md 中重复的 #5412/#5415 两行,共 3 行)。逐项核实:

  • 远端 head 与上一轮完整审过的内容逐字节一致(对本地审查分支 git diff 为空);
  • 相对 public master 的真实 diff = 9 文件 +1847/−193(即上轮 +1850/−193 减去这 3 行),无任何其他变化;
  • CHANGELOG.md 中 #5412 / #5415 各恰好出现一次;
  • 作者此后无新提交、无新讨论。

八问(简版,完整分析见上方首轮 review)

  1. 是什么问题:共享 state 目录快照(prefix-affinity / session-order)并发写丢更新的根治(#1724 家族)。
  2. 什么原因:可变单文件 + 整文件替换的发布模型没有跨进程互斥,两个陈旧视图互相覆盖。
  3. 影响与严重度:仅多实例共享 state 目录;特定条件下丢亲和链/会话顺序(前缀缓存退化),不崩溃不阻断。
  4. 怎么修:不可变发布日志——staging 写满 fsync → 唯一 pub-* 目录原子 rename → 父目录 fsync → 仅被证明覆盖且字节复核仍在盘上的输入才回收;读取端取存活叶与冻结基线并集。
  5. 回归:合并规则/深度优先/LRU 1024/8MiB 上限/去抖时序逐行未变;wire/配置面/env/kernel/ /update.ts/version 未触碰;有意行为变更(落盘布局、损坏处理、非耐久 FS 累积、tie 方向)已在首轮正文逐条披露(旧值→新值)。
  6. 缓存:不破坏;多实例下反而消除互相覆盖引起的前缀缓存抖动。
  7. 设计方向:局部转向「可变电文 → 不可变日志 + 冻结基线」,与仓库账本式身份哲学一致,纯持久化层内部。
  8. 新配置:无;磁盘布局变化(备份需含 .journal/)已披露。

CI

镜像缓存 @ head a3d6a94:30 项检查全部 completed,0 failed,0 pending——绿。

结论:可合。 唯一发现(changelog 重复条目)已顺手修掉并验证,CI 绿,待人工合并。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

复审完成(新 head a3d6a94):APPROVE 已提交(review 5800098467)。核实分支自上一轮批准以来仅含本车道顺手修复 commit a3d6a94(CHANGELOG 去重 3 行),与审过内容逐字节一致,作者无新增;镜像缓存 30 项检查全绿(0 failed / 0 pending);无 kernel/ 触碰、无需决策。PR 现无阻塞项,待人工合并。

@ranxianglei ranxianglei left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(ai: vllm-qwen/qwen3.8-27b)

Round-3 re-review — APPROVE. The only change since the last approved head (a3d6a94, reviews 5799891897 / 5800098467) is a single merge commit 0f0b59f that syncs the branch with current master tip 406193b. No new author commits, no file edits, no PR-body changes. Verified against the live tree:

  • Structure: git log -1 --format=%P → parents a3d6a94 (approved head) + 406193b (current public master tip, fetched independently). First-parent chain confirms the branch adds nothing beyond that merge.
  • True PR delta vs current master: exactly the same 9-file feature set, +1847/−193 — byte-for-byte the content reviewed and approved in rounds 1–2.
  • Merge integrity: the six core files (src/snapshot-journal.ts, src/affinity-persist.ts, src/order-persist.ts, tests/snapshot-journal.test.ts, tests/affinity-persist.test.ts, tests/issue3397-persona-restart-stability.test.ts) are byte-identical to the approved head; the three files master also touched (CHANGELOG.md, reference/architecture.md, scripts/live-files.mjs) merged cleanly — all four changelog entries, the architecture row, and the live-files entry survived, and #5412/#5415 each appear exactly once (the round-1 dedup fix held through the merge).
  • Import sources moved on master: src/prefix-affinity.ts carries a comment-only doc-path update; src/session-resolve.ts has no export/signature change to the symbols order-persist consumes.
  • Authorization: PR's own delta touches no kernel/; no kernel-change declaration in the body; human-tier gate not triggered.
  • CI at new head: mirror cache shows 30 checks, 0 failed (6 success + 11 skipped completed, 13 still running at review time); lane reports green.

结构化八问

  1. 是什么问题:两个共享同一 state dir 的实例对亲和链/会话顺序快照做进程本地读改写,唯一 tmp 名也挡不住 stale-view 丢失更新;本 PR 用不可变发布日志(私有暂存→fsync→原子 rename 到唯一发布名→按覆盖+字节复核回收输入)根治。详见 R1/R2 完整分析。
  2. 什么原因导致的:旧路径共享 <name>.json.tmp + 进程内 RMW 协调,两个 stale 视图先后落盘即丢更新;根因是「可变单一文件 + 非原子读改写」,不是 tmp 命名。
  3. 影响范围与严重程度:多实例同机部署(dsh profile/多宿主)下的数据静默丢失;单实例无感。属 p2 级正确性缺陷,修复后消除。
  4. 怎么修复的:不可变发布 + 覆盖门控回收(covers(output,input) 对该叶 census 私解析数据判定,字节复核后才 unlink)+ 耐久门控(目录 fsync 失败则零回收、如实报 uncovered)+ 损坏/半删输入永久保留(sticky incomplete)。
  5. 是否有回归问题:R1 已逐条清点既有行为——合并规则/深度优先/LRU 1024/字节上限/500ms 去抖 5s 最大延迟全部未变;tie 方向翻转(mine-wins→disk-wins)实际不可观测;损坏处理从「过滤+flush 自愈」改为「整份保留」是有意的守恒取舍,已披露并有测试钉死。本轮同步合并未引入任何新行为变化。
  6. 是否破坏缓存:不涉及(prefix cache/重试/游标均不受影响;快照文件不是缓存层)。
  7. 是否改了设计方向:持久化布局变更(legacy 基线冻结 + .journal/ 不可变叶)是披露过的有意变更,备份须含 .journal/;非架构转向。
  8. 是否引入了新的配置:无新 env/config/默认值。

平台限制(非耐久 FS 上叶目录累积、sticky incomplete 告警噪音、Windows/网络文件系统不承诺)在 PR 正文、CHANGELOG 与模块文档中均有披露并由测试钉住,属作者明确接受的边界,不再作为待办项重列。

Verdict: APPROVE — the PR's contribution is unchanged from the twice-approved content; the new head is a clean sync with current master. Merge remains human-only.

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

Round-3 review complete (new head 0f0b59f): single master-sync merge on top of the approved a3d6a94 — no new author commits; true PR delta vs current master is byte-for-byte the twice-approved 9-file set (+1847/−193), all three overlap files merged cleanly (#5412/#5415 dedup held); no kernel/ touch, nothing to fix or decide; CI at new head: 0 failed (13 checks still running at review time, lane reports green). APPROVE posted — review 5824803127. Awaiting human merge.

@ranxianglei
ranxianglei marked this pull request as ready for review November 10, 2026 01:29
Resolve the ranxianglei#5480 (immutable snapshot journals) vs master (ranxianglei#5753 kv_store
unified stores) conflict by union of intent:

- src/affinity-persist.ts / src/order-persist.ts: keep BOTH the ranxianglei#5480
  census/publish writer (under master's writeSnapshotLegacy call-site name)
  and master's kv machinery; re-add the readDiskEntries/mergeWithDisk
  disk-view helpers master's kv path calls, reworked journal-aware (baseline
  ∪ published leaves) because ranxianglei#5480 never rewrites the baseline file again.
- tests/*-unified-store.test.ts: the pre-hydration fallback now publishes a
  durable journal leaf instead of rewriting the legacy file — assert the
  leaf, not the baseline's existence.
- CHANGELOG.md / scripts/live-files.mjs / reference/architecture.md: union
  of both sides' entries.
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Conflict resolved and pushed. Merged master (a52c300) into the PR branch; resolution commit c49a46def3 is now the head of Maziar123:fix/concurrent-snapshot-persistence (fast-forward from 0f0b59f0c4, plain push — no force-push, no other agents were on the branch: remote head verified unchanged before pushing).

Why it conflicted: since this PR branched, master landed #5753 ("move six JSON sidecar states onto the unified kv_store"), which rewrote the same regions of src/affinity-persist.ts / src/order-persist.ts that #5480 rewrites (raw file writer → immutable journal publications). Four files conflicted: those two modules plus CHANGELOG.md and scripts/live-files.mjs.

Resolution — union of intent, no behavior unification:

  • Both persist modules keep BOTH mechanisms: the PR's census/publish journal writer (under master's writeSnapshotLegacy call-site name) AND master's kv_store unified-store machinery.
  • Master's new kv-path callers reference readDiskEntries() / mergeWithDisk(), which the PR had deleted — the auto-merge silently dropped the definitions while keeping those callers (would have been a compile break). Re-added both: mergeWithDisk verbatim from master; readDiskEntries reworked journal-aware — it returns journal.census([]).records (baseline ∪ published leaves) instead of reading only the baseline file. Rationale: after [PR] fix(persistence): publish concurrent snapshots through immutable journals #5480 the baseline file is never rewritten again, so a baseline-only reader would resurrect exactly the lost-update class [PR] fix(persistence): publish concurrent snapshots through immutable journals #5480 fixes at the file→kv migration boundary (leaf-only chains published before hydration would be permanently invisible to unified instances).
  • Kept BOTH mergeEntries and mergeWithDisk in each module — their tie-breaks deliberately differ; unifying them would be an unrequested behavior change.
  • CHANGELOG.md + scripts/live-files.mjs: union of both sides' entries (both entries kept in the spawn-tripwire list).
  • Test adaptation (disclosed): master's Existing JSON State Files: Inventory and First-Batch Unified Storage Migration | 存量 JSON 状态文件:盘点与第一批统一存储迁移 #5753 suites tests/affinity-unified-store.test.ts / tests/order-unified-store.test.ts asserted that the pre-hydration fallback rewrites the legacy JSON file. Post-[PR] fix(persistence): publish concurrent snapshots through immutable journals #5480 that write goes through the immutable journal instead — the baseline is never touched, and that immutability IS the fix. The two assertions now check for a published durable journal leaf beside the file (via journalDirOf); the "hydration then imports it" half of each test is unchanged and passes.

Verification (local, targeted): tsc --noEmit clean (exit 0); all six suites importing the touched modules pass — 55/55 (snapshot-journal, affinity-persist, order-unified-store, affinity-unified-store, issue3397-persona-restart-stability, responses-previous-response-id). Full-suite verdict remains remote CI's job.

CI state (live-checked via mirror cache): the old head 0f0b59f0c4 ran fully green (30 checks, 0 failed) — the only blocker was the conflict itself, there were no red checks to fix. The mirror cache still served the pre-push snapshot during this run (its fetch timestamp predates the push), so I cannot yet report the new head's check runs or the recomputed merge state from a live source — they will appear on the next mirror poller cycle. By construction the branch now carries master as a parent, so merge state should clear once GitHub recomputes.


中文八问(审核清单)

  1. 是什么问题:PR 分支与 master 合并冲突(merge state dirty)——master 的 Existing JSON State Files: Inventory and First-Batch Unified Storage Migration | 存量 JSON 状态文件:盘点与第一批统一存储迁移 #5753(kv_store 统一存储)和 PR 的 [PR] fix(persistence): publish concurrent snapshots through immutable journals #5480(不可变快照日志)重写了同一片持久化代码。
  2. 什么原因导致的:两条线各自替换了同一批函数(PR 把裸文件写盘换成 journal 发布,master 在同一区域加了 kv 优先/legacy 回退);文本冲突之外还有语义耦合——master 的新调用方引用了 PR 已删除的 helper(自动合并会静默留下编译断裂)。
  3. 影响范围与严重程度:仅该 PR 分支不可合并,不影响任何已发布版本;不解决则 PR 永远无法合入。旧 head 的 CI 本身是全绿的,唯一阻塞就是冲突。
  4. 怎么修复的:merge master 进 PR 分支(不 rebase,沿用仓库先例),按「意图并集」解冲突——两套机制都保留;共用磁盘视图 helper 按 [PR] fix(persistence): publish concurrent snapshots through immutable journals #5480 新现实重写为 census(baseline ∪ journal 叶);Existing JSON State Files: Inventory and First-Batch Unified Storage Migration | 存量 JSON 状态文件:盘点与第一批统一存储迁移 #5753 两个测试断言从「回退重写 legacy 文件」改为「回退发布 durable journal 叶」。
  5. 是否有回归问题:审过既有行为——kv 迁移路径、legacy 回退路径、hydrate 导入、防抖写盘均保留原语义;唯一行为变化是「预 hydrate 回退写的落点」从 legacy JSON 文件本身变为旁边的 journal 叶——这是 [PR] fix(persistence): publish concurrent snapshots through immutable journals #5480 的预期设计(baseline 不可变正是修复本体),非回归;本地 tsc 干净 + 相关 6 套件 55/55 全绿,全量回归以远程 CI 为准。
  6. 是否破坏缓存:不涉及(前缀缓存/重试/游标均未触碰;磁盘读取视图只变得更完整)。
  7. 是否改了设计方向:不改方向——[PR] fix(persistence): publish concurrent snapshots through immutable journals #5480 与 Existing JSON State Files: Inventory and First-Batch Unified Storage Migration | 存量 JSON 状态文件:盘点与第一批统一存储迁移 #5753 合并后共存:journal 管文件侧不可变发布,kv_store 管统一存储侧。
  8. 是否引入了新的配置:不涉及(无新 env/config/默认值)。

判定:冲突已解决并推送(c49a46d),等待 GitHub 对新 head 的重算与 CI 复跑(镜像缓存本周期尚未刷新到新 sha)。

…d pass)

Master moved +100 commits past c49a46d; conflicts were import hunks only
in src/affinity-persist.ts and src/order-persist.ts (both sides added
different imports at the same spot). Resolved as union: openSnapshotJournal
(ranxianglei#5480) + persistDebounceMs/persistMaxDelayMs (ranxianglei#6058/ranxianglei#6062) + kvStore (ranxianglei#5753).
Master's ranxianglei#6058 change (hardcoded PERSIST_DEBOUNCE_MS/PERSIST_MAX_DELAY_MS
constants replaced by the shared persist.* knobs in schedule*Persist)
auto-merged clean on top of the journal writers.

tests/affinity-persist.test.ts + tests/order-persist.test.ts: master's new
ranxianglei#6058 knob timing tests asserted the legacy baseline file appearing on disk;
since ranxianglei#5480 the legacy writer publishes immutable journal leaves and never
rewrites the baseline, so those assertions could never hold here. Adapted the
landing checks to journal pub-* leaves (same disclosed adaptation class as
c49a46d's unified-store test fixes). The knob wiring under test is
unchanged; the tests now pin it through the real publication path.
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Second conflict pass done and pushed: c49a46def3..bd5c613029 (plain fast-forward, no force-push).

What the live check found this run. Yesterday's merge commit c49a46def3 was still the PR head (no concurrent pushes on the branch), but master had moved +100 commits in the meantime and re-touched both persistence modules — git merge-tree against freshly-fetched refs confirmed a real new conflict before I touched anything (not a stale trigger): a7d9f5dcb4 (#6058/#6062) replaced the hardcoded PERSIST_DEBOUNCE_MS/PERSIST_MAX_DELAY_MS constants in src/affinity-persist.ts and src/order-persist.ts with the shared persist.* knobs, and its new import landed exactly where #5480 had just added the journal import.

Resolution (union of intent). Merged origin/master@58a8796224 into the branch. Conflicts were the two import hunks only — kept all three imports (openSnapshotJournal #5480 + persistDebounceMs/persistMaxDelayMs #6058 + kvStore #5753). Master's knob rewiring auto-merged clean on top of yesterday's journal writers; the rest of the 100 commits (docs, other modules) came through untouched.

Stale assertions caught and fixed (disclosed adaptation). Master's new #6058 knob timing tests — 2 added to tests/affinity-persist.test.ts, a new tests/order-persist.test.ts (3 tests) — assert that the legacy baseline JSON file appears on disk after a write. Since #5480 the legacy writer publishes immutable journal leaves and never rewrites the baseline, so those assertions could never hold on this branch (they'd red-CI the moment they ran). I adapted the landing checks to judge pub-* journal leaves instead (the affinity file already had a snapshotLanded() helper; the order file got an equivalent). The knob wiring under test is unchanged — the tests now pin it through the real publication path.

Verification. tsc --noEmit exit 0; marker sweep clean; targeted suites (snapshot-journal, affinity-persist, order-persist, affinity-unified-store, order-unified-store, issue3397-persona-restart-stability, responses-previous-response-id) = 60/60 pass, including all five adapted knob tests. Full-suite verdict belongs to remote CI on the pushed head.

CI state, honestly. The mirror-cache entry for this PR is stale (fetched_at ~30h ago; last showed the previous head c49a46def3 as mergeable-state behind, 0 failed checks). The new head bd5c613029 was NOT observable in any live source during this run, so I'm not asserting its check results. By construction the branch now fully contains master@58a8796224, so no conflict can exist until master moves again.


中文八问(供人工审核)

  1. 是什么问题:PR 与 master 再次冲突——24 小时内 master 前进了 100 个提交,billion-context Physical Write Volume on QLC SSD: 31-Day Field Test, Four Governance Tiers, Residual Attribution | billion-context 在 QLC SSD 上的物理写入量:31 天实测、四道治理、剩余归因 #6058/Time-context snapshots injected as role:"user" cause spurious "fake user messages" #6062 重写了两个持久化模块的防抖常量;且 master 新增的 5 个 knob 计时测试断言的是「基线文件被重写」,而 [PR] fix(persistence): publish concurrent snapshots through immutable journals #5480 恰恰把该行为改成了「发布不可变日志叶」,这些测试在本分支上必然失败。
  2. 什么原因导致的:热文件竞争。billion-context Physical Write Volume on QLC SSD: 31-Day Field Test, Four Governance Tiers, Residual Attribution | billion-context 在 QLC SSD 上的物理写入量:31 天实测、四道治理、剩余归因 #6058 把硬编码常量换成共享 persist.* knob,其新 import 恰好落在 [PR] fix(persistence): publish concurrent snapshots through immutable journals #5480 刚加 journal import 的同一位置(文本冲突);新测试按 [PR] fix(persistence): publish concurrent snapshots through immutable journals #5480 之前的写盘行为编写,与不可变日志发布机制不兼容(断言失效)。
  3. 影响范围与严重程度:仅影响本 PR 的可合并性与后续 CI;无生产行为影响。若不处理则 PR 无法合并;断言问题本身不影响运行时。
  4. 怎么修复的:merge master 进分支;两处 import 冲突按并集解决(三条 import 全保留);5 个 knob 计时测试的「落盘判定」从基线文件存在改为 journal pub-* 叶存在——只改断言载体,被测的 knob 接线逻辑一行未动。
  5. 是否有回归问题:逐项审过——昨日落地的 journal 发布路径(census/publish/writeSnapshotLegacy)字节级未变;billion-context Physical Write Volume on QLC SSD: 31-Day Field Test, Four Governance Tiers, Residual Attribution | billion-context 在 QLC SSD 上的物理写入量:31 天实测、四道治理、剩余归因 #6058 的 knob 调用干净自动合入 schedule*Persist;两条 kv_store 统一存储路径未变;默认节奏仍是 500ms/5s(knob 默认值保持历史值)。无语义漂移。
  6. 是否破坏缓存:不涉及。未触碰缓存/重试/增量游标/去重键;仅防抖窗口取值来源变化(硬编码→knob,默认值不变)及测试断言载体变化。
  7. 是否改了设计方向:没有。这是第二次接冲突,三个机制([PR] fix(persistence): publish concurrent snapshots through immutable journals #5480 不可变日志、Existing JSON State Files: Inventory and First-Batch Unified Storage Migration | 存量 JSON 状态文件:盘点与第一批统一存储迁移 #5753 统一存储、billion-context Physical Write Volume on QLC SSD: 31-Day Field Test, Four Governance Tiers, Residual Attribution | billion-context 在 QLC SSD 上的物理写入量:31 天实测、四道治理、剩余归因 #6058 共享 knob)的方向全部保留并按并集共存。
  8. 是否引入了新的配置:本人未引入。master 侧 billion-context Physical Write Volume on QLC SSD: 31-Day Field Test, Four Governance Tiers, Residual Attribution | billion-context 在 QLC SSD 上的物理写入量:31 天实测、四道治理、剩余归因 #6058 带来 BILI_PERSIST_DEBOUNCE_MS/BILI_PERSIST_MAX_DELAY_MS(默认 500ms/5s 不变),属合并带入而非本次新增。

结论:冲突已解、已推送 bd5c613029,本地定向验证全绿;等新 head 的 CI 重算结果。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[ework] CI fix verified green ✓

@ranxianglei ranxianglei left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(ai: vllm-qwen/qwen3.8-27b)

Round-4 review — head bd5c613 (two master-sync merges on top of the approved 0f0b59f).

What changed since round 3. No new feature commits. c49a46def (first pass) and bd5c613 ("second pass") merge public master (58a8796, current tip) into the branch. Master had meanwhile changed the exact modules this PR rewrites — #6062 wired the new persist.debounceMs / persist.maxDelayMs knobs, and #5753 moved both snapshots onto the unified kv_store carrier (the file becomes the pre-hydration fallback / legacy-only / post-demote path, with one-time migration and writeSnapshot → writeSnapshotLegacy). These merges are therefore a semantic union of two features touching the same functions, not mechanical catch-up — reviewed as such: both merged source files were read in full, plus all three adapted test files.

Verdict on the union: clean.

  • src/snapshot-journal.ts, tests/snapshot-journal.test.ts, tests/issue3397-persona-restart-stability.test.ts: byte-identical to the twice-approved content.
  • src/affinity-persist.ts / src/order-persist.ts: the journal mechanism (immutable publication, coverage-gated reclaim, durability gate, conservation-law parse) is intact line-for-line; master's kv carrier layers on top without touching it. Routing in both schedule* and flush* is kvReady && kvStore() !== null → unified chained RMW, else the journal file path — which preserves the #5548 carrier rule (a hydrated unified instance never writes the file again), so the pre-kv fallback is exactly where the journal now publishes. The deliberate parser asymmetry survives too: the kv-row parser filters invalid entries (row rewritten wholesale each time) while the file/journal parser throws (retention law — a later publish must not be able to reclaim records a corrupt input still holds).
  • Debounce wiring uses the new knobs (min(persistDebounceMs(), persistMaxDelayMs() − elapsed)); defaults keep the historical 500 ms / 5 s. Those knobs arrived via master's #6062 — inherited by merge, not introduced by this PR.
  • Test adaptations (tests/order-persist.test.ts, tests/affinity-unified-store.test.ts, tests/order-unified-store.test.ts): assertions repointed from "legacy file exists" to "a pub-* leaf exists under <name>.json.journal/", with a comment explaining why (the baseline is never rewritten). Timing semantics (default cadence, knob windows) are unchanged; the late-hydration-import assertions now additionally prove that migration reads the journal union. No assertion was weakened, and none still asserts the old rewrite/heal behavior.
  • Docs: all four CHANGELOG entries survive verbatim; the #5412/#5415 dedup held through both merges (one occurrence each); the reference/architecture.md row and the scripts/live-files.mjs spawn-tripwire entry are present.
  • No kernel/ files in the PR delta, no kernel-change declaration → no authorization boundary engaged.

CI. Mirror cache at bd5c613: 30/30 completed, 0 failed.

APPROVE. Merge remains human-only.

中文八问

  1. 是什么问题:多个并发持久化进程会互相覆盖快照文件(丢更新);本 PR 用不可变发布日志替代共享临时文件名方案。
  2. 什么原因导致的:旧实现所有进程写同一个 <name>.json.tmp 再 rename,后写者用自己的陈旧视图合并结果覆盖先写者的内容;进程内协调挡不住跨进程竞态。
  3. 影响范围与严重程度:多实例/重启竞态下亲和链与会话顺序记录丢失(#1724/#3397 家族);单实例单进程用户无感。数据完整性缺陷,非可用性缺陷。
  4. 怎么修复的:私有暂存目录写满并 fsync 负载与目录 → 原子 rename 到唯一 pub-* 名 → fsync 父目录;回收仅在「已落盘输出逐键覆盖该叶自解析数据」且目录屏障成功时发生;损坏/不可读输入原样保留并报告;读端取全部有效叶与冻结基线的并集(沿用原合并策略)。本轮新增面:与 master 同期落地的 kv 载体/去抖旋钮特性做语义合并——kv 水合后走统一存储,未水合或降级时才走本日志文件路径,两套机制互不干扰。
  5. 是否有回归问题:已审既有行为——合并规则、深度优先、LRU 1024 上限、字节上限、500ms/5s 去抖节奏(现经旋钮解析,默认值未变)、水合语义均保持;损坏文件处理从「过滤无效项+下次落盘自愈」改为「整份保留、不再自愈」是有意变更(守恒优先,已在 CHANGELOG/PR 正文披露并有回归钉住);非持久文件系统上 pub-* 叶只增不减为作者披露的已知边界(失败屏障诚实性有故障注入测试)。无意外回归。
  6. 是否破坏缓存:不涉及模型前缀缓存;会话状态文件的读取方(同步水合、kv 一次性迁移)全部改走 census 并集,迁移断言已验证早期写入仍能被导入。
  7. 是否改了设计方向:磁盘布局由「单文件原地改写」转为「冻结基线 + 不可变发布日志」,备份需连同 <name>.json.journal/ 一起处理;这是本 PR 的核心设计变更,已披露并由回归测试钉住。
  8. 是否引入了新的配置:本 PR 自身未引入新配置;persist.debounceMs / persist.maxDelayMs(默认 500ms/5s,保持历史值)来自 master #6062,经合并继承。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

🤖 Powered by ework · qwen3.8-27b

Round-4 review complete (head bd5c613): the two new commits are master-sync merges only, and master had meanwhile rewritten the same modules (#6062 persist knobs + #5753 unified kv carrier) — so I re-reviewed the semantic union in full rather than rubber-stamping: journal mechanism intact line-for-line, kv carrier layered correctly on top (pre-kv fallback = exactly where the journal publishes), three master test files correctly adapted to publication semantics (no weakened assertions), docs/dedup survived both merges, no kernel/ touch, CI 30/30 green. APPROVE posted (review 5853001448); merge remains human-only.

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[ework] CI fix verified green ✓

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Third trigger arrived claiming merge-state "dirty" — live verification this run found no real conflict and no red CI, so I took no action on the branch. Evidence, all gathered fresh in this run:

  1. Heads: fetched both remotes first. PR head is still bd5c613029 (yesterday's second-pass merge) — no concurrent pushes on the branch. Master moved +37 commits since then (58a8796224 → 33104b1f95).
  2. Deterministic conflict check: git merge-tree --write-tree of current master into the PR head → exit 0, zero conflicts. The branch merges cleanly into today's master.
  3. Mirror cache (refreshed this run): sha bd5c613029, mergeable_state = behind (not dirty), all 30 checks green, 0 failed, 0 pending on the pushed head.

What master's +37 did to the touched files: one commit (851e3bb766, "settle and reset unified snapshot write batches") made persistUnifiedNow() async in both modules — that's the #5753 unified-kv path, which #5480 deliberately leaves alone; I checked the hunks for semantic interaction with yesterday's resolutions (journal writer, readDiskEntries, import union) and found none. It also merged textually clean, which is why merge-tree is green.

Why I didn't merge master again: "behind" is not a merge blocker (GitHub re-merges head into whatever master is at merge time, and that merge is provably clean right now), master has moved ~137 commits across these two days so chasing it is whack-a-mole until a human merges, and piling another merge commit onto an external contributor's PR adds noise without changing mergeability. If a genuinely dirty state or a red check appears, the next trigger gets a real pass.

The PR is ready for human review/merge.


中文八问(供人工审核)

  1. 是什么问题:无实际问题——这是一次基于过期状态的重触发。活体验证确认 PR 当前为「behind」而非「dirty」,且新 head 上 CI 全绿。
  2. 什么原因导致的:触发管道的 merge-state 数据滞后于 GitHub 重算(上一轮已把冲突修掉并推送);master 持续前进(本轮又 +37 提交)。
  3. 影响范围与严重程度:无影响。不影响生产、不影响可合并性(当前即可合并)。
  4. 怎么修复的:无需修复。先用三个独立来源验证(远端 refs 比对、merge-tree 干跑、镜像缓存检查态),确认无可执行阻塞后才决定不动分支——不追加多余的 merge commit。
  5. 是否有回归问题:本轮零代码改动,无回归引入。另审了 master 新变更(persistUnifiedNow 异步化,Existing JSON State Files: Inventory and First-Batch Unified Storage Migration | 存量 JSON 状态文件:盘点与第一批统一存储迁移 #5753 统一存储路径)与 [PR] fix(persistence): publish concurrent snapshots through immutable journals #5480 解决方案的语义交互:无交集。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:没有。
  8. 是否引入了新的配置:没有。

结论:PR bd5c613029 冲突不存在、CI 全绿、可合并;等待人工 review/merge。若再出现真实冲突或红 CI,下轮触发会实际处理。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

[ework] CI fix verified green ✓

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants