Skip to content

[PR] fix: MySQL v8 upgrade cleans up duplicate message rows left by wireIndex splits (#4666) - #4678

Merged
ranxianglei merged 6 commits into
masterfrom
2026-10-25_mysql-v8-dedup
Oct 27, 2026
Merged

ranxianglei merged 6 commits into
masterfrom
2026-10-25_mysql-v8-dedup

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Oct 25, 2026 •

Copy link
Copy Markdown
Owner

Fixes #4666 · family: #4806 (MySQL OOM facet), template: #4677/#4684 (sqlite streaming)

What this PR does

Legacy MySQL libraries upgraded to v8 keep their pre-#4390 wireIndex-split duplicates forever: several rows per logical message (same msg_id, different content_hash, identical identity once id/wireIndex are stripped). The sqlite v8 migration collapses each (session_id, msg_id, identity) group to its lowest seq and refunds the dropped rows' CAS refs; the MySQL migration backfilled identity but skipped the collapse — affected libraries replay every duplicate forever, and #4390's historian sweep can never remove them (both rows "hit" the incoming identity set).

This PR gives the MySQL upgrade the same collapse step, built as the sqlite lane's streaming design (#4677) plus a cross-process lock:

  1. Streaming data step (migrateV8Identities, src/storage/mysql.ts): sessions selected by identity = '', ONE SESSION PER TRANSACTION over 500-row seq-cursor pages; dedup happens inside each session's ordered scan (first arrival kept, later generations deleted with media + text CAS refs refunded, refcount > 0 floor); rows whose content bytes are missing from the CAS get the printable absent: marker (byte-parity with schema.ts); progress log every 100 sessions. Memory holds one page window + one session's keeper keys — never the whole table's bytes (MySQL storage backend v8 backfill loads the whole table into memory at once and skips dedup | MySQL 存储后端 v8 回填一次性加载全表且不按 msg_id 去重 #4806 class).
  2. Stamp-gated resume: the data step runs only while bili_meta schema_version < 8 (NOT column presence); the stamp advances to 8 only after all sessions commit; a crash mid-migration resumes from committed sessions on next boot.
  3. Cross-process named lock: GET_LOCK('bili_v8_upgrade_<db>') around the data step — connection-scoped (auto-released if the migrator dies), per-database scoped, waiters retry until acquired. Two instances booting against the same library can no longer double-delete dup groups / double-refund refs.
  4. Same-PR facets recorded in Duplicate message rows not cleaned up after upgrading legacy MySQL to v8 (the SQLite version did clean them): migration | MySQL 老库升 v8 后重复消息行没被清理(sqlite 版清了):迁移少一步去重,回放重复且永不自愈 #4666: sqlite decRef gains the refcount > 0 floor; the redundant second identity pass is gone; verifyProjections checks RETIRED rows on both backends (diagnostic coverage gap).
  5. CI: the mysql-contract job also runs tests/storage-backend-mysql-e2e.test.ts, which documents itself as running there but was dead-listed (separate chore commit).

Review rework (owner review of this PR, 2026-10-27)

Item Status
① rebase onto the #4677-streaming master c5fedde (conflict lane, union-of-intents resolution)
② full-table load → per-session streaming 6937a1d
③ multi-instance double-delete window 6937a1d (server-side named lock)

Behavior changes (old → new)

Surface Old New Why
MySQL v7→v8 upgrade backfill identity only; duplicates persist forever collapse each (session_id, msg_id, identity) group to lowest seq, refund CAS refs the fix itself (#4666)
MySQL upgrade memory footprint whole messages×content join materialized in the Node heap one 500-row page window + one session's keeper keys #4806 OOM class
concurrent MySQL upgrades unsynchronized (double-delete / double-refund race) serialized by a server-side named lock; waiters retry until acquired data-safety race (review item ③)
MySQL upgrade crash recovery stranded at identity='' forever (column-presence gate) resumes from committed sessions (stamp gate); stamp advances last crash safety
MySQL CAS-missing rows during migration (path did not exist) printable absent: marker, parity with sqlite session termination guarantee
SQLite migration decRef bare refcount-1 (corrupt ledger could go negative) floored at 0 via refcount > 0 strictly safer; gc collects <= 0 anyway
verifyProjections (both backends) live rows only includes retired rows diagnostic coverage gap (issue facet ③)
CI mysql-contract job storage-backend-mysql-e2e.test.ts dead-listed runs it suite header claims it runs there

No config-surface change. No kernel/ change. No version bump.

Tests

tests/issue4666-migration-dedup.test.ts (dual-lane; mysql lane runs in the CI mysql-contract job against throwaway databases):

  • sqlite: collapse + refund + floor pins; verifyProjections retired-row drift pin.
  • mysql: fresh install stamps 8; downgrade surgery + dirty fixture → collapse/refund/floor/replay byte-exact; crash arm (stamp 7 + unbackfilled pair + CAS-missing row) → resume with byte-exact absent-marker; cross-page-boundary dedup (503-row session, dup generation sits past the 500-row seam); concurrent-boot serialization (second connection holds the lock → migrator waits with fixture untouched; data planted behind the lock is still processed ⇒ enumeration happens after acquire; exactly-once collapse).

@ranxianglei ranxianglei changed the title fix: MySQL v8 upgrade cleans up duplicate message rows left by wireIndex splits (#4666) [PR] fix: MySQL v8 upgrade cleans up duplicate message rows left by wireIndex splits (#4666) Oct 25, 2026
@github-actions

github-actions Bot commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-25_mysql-v8-dedup (42aad38)

Latest build — install this exact version

npm install -g billion-context@0.0.6401-pr.4678.fc0cfb4

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
42aad38 billion-context@0.0.6401-pr.4678.fc0cfb4 build
a476a90 billion-context@0.0.6395-pr.4678.c4312ad build
625a68b billion-context@0.0.6386-pr.4678.44850c4 build
6937a1d billion-context@0.0.6382-pr.4678.292879a build
c5fedde billion-context@0.0.6369-pr.4678.6f53994 build
f64f231 billion-context@0.0.5718-pr.4678.cbcebc0 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr4678.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

查重: 不重复。 #4678 是 #4666(MySQL v8 迁移缺去重)的唯一在途 PR;open 侧无同症状/同机制条目。相邻的 #4672(PR for #4665,archive/restore 的 identity+retired_at 携带)是不同失败面(#4390 身份清扫 vs 本 PR 的 v8 迁移去重),互不覆盖。

状态:agent 已完工 PR(首行 ai 自声明),我按「识别自己的 PR」流程处理——不重做,拉分支核对代码 + 确认 CI。当前 CI:静态门禁全绿(kernel-guard / config-surface-guard / lane-guard / version-guard 等已完成 success),test shards + mysql-contract in_progress。拉分支审查中,结论稍后按八问体例补上。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

#4678 独立审核完成(未照抄 issue 结论,diff 逐段重读、机制逐条重验)。查重已在首条回复给出:不重复。

审核做了什么:完整读了 5 个文件的 diff 与相关既有代码(identity.ts、store.ts retire/sweep、mysql.ts tx/getMessages、schema.ts migrateSchema、CI job 定义、新测试套件全文);验证了崩溃矩阵、幂等性、调用者面;本地在 PR head worktree 跑了新套件(sqlite lane)+ 6 个与 diff 直接耦合的存储套件 + typecheck;CI 红片做了分片复算归因。

结构化八问

  1. 是什么问题:MySQL 库从 v7 升到 v8 时只回填了 identity 列,从不合并重复行——带 wireIndex 分裂多代(同 msg_id、多行、不同 content_hash)的旧库会永久回放重复消息,historian 清理也删不掉。
  2. 什么原因导致的:两个后端的升级路径不对称——sqlite 的 v8 迁移一直有「按组塌缩到最低 seq + 退还 CAS 引用」,MySQL 侧只有回填;而清理机制以 (msg_id, identity) 为键,同一逻辑消息分裂出的所有代都命中同一 incoming 集合,永远无法互相淘汰。修升级路径是唯一可行的修复点。
  3. 影响范围与严重程度:仅存量 MySQL 库中经历过 wireIndex 分裂的会话;症状是重复行永久回放(上下文膨胀、token 持续浪费),可用但随时间恶化;sqlite 用户与新安装不受影响。
  4. 怎么修复的:把 MySQL 升级对齐 sqlite 契约——probe-first ALTER 之后,数据步骤在单个事务内对每个 (session_id, msg_id, identity) 组保留最低 seq 行,删除其余行并退还 media/text CAS 引用(带 refcount > 0 下限)。关键设计:数据步骤用 bili_meta.schema_version 戳门控而非列存在性(MySQL DDL 自动提交,崩溃在 ALTER 与数据步骤之间时下次启动幂等重跑),戳用 GREATEST 保证只升不降。同 PR 顺带三个 facet:sqlite 迁移的裸 decRef 加下限、删掉冗余的第二遍 identity 计算、verifyProjections 覆盖 retired 行;另外把自称跑在 mysql-contract job 却从未被列入文件清单的 e2e 测试真正接进 CI。
  5. 是否有回归问题(逐项审过既有行为):
    • 新装路径:DDL 本就含 v8 列 → 空事务 + 戳 '8',行为不变 ✅
    • 升级幂等:二次运行每组只剩 keeper 行,UPDATE/DELETE 全为 no-op ✅(测试有 crash arm 钉住)
    • getMessages 默认形状不变:includeRetired 是既有可选参数,默认仍排除 retired ✅
    • verifyProjections 收紧:全仓零生产调用者(仅接口声明/facade 透传/测试),纯诊断面;retire 是 tombstone-only(text CAS 引用有意不释放,投影保持有效)→ 不会误报,测试双向钉住 ✅
    • sqlite 迁移:除删死代码 + 加外,语义不变,严格更安全 ✅
    • CI job:只在既有 run 行追加两个文件 ✅
      无 fail-fast→吞错、无阈值/时序/输出格式漂移。PR 披露的行为变更表 4 项我逐条对照代码核实,与 diff 一致。
  6. 是否破坏缓存:不涉及(存储层);CAS 退还镜像 releaseContent,gc 收集 <=0,下限防迁移中途负数。
  7. 是否改了设计方向:无转向——让 MySQL 升级路径对齐既有 sqlite v8 契约,未引入新架构。
  8. 是否引入新配置:无(无新 env/config/CLI)。

测试质量:断言有牙——缺去重则 survivors=8≠4、缺退还则 refcount=1≠0、缺下限则 -1≠0,三条都能抓住原 bug;crash arm 从半迁移态(戳 '7'、列已存在、identity='' 行)重启,直接区分「戳门控」与「列存在门控」;retired-row 校验先钉干净通过再钉损坏失败,双向都有。小瑕疵(不阻塞):swap 分支(r.seq < seen.seq)无测试覆盖——在 ORDER BY + 唯一 (session_id,seq) PK 下是死分支,属防御性 parity 代码。

CI 状态:7 个红 test 片经核对均非本 PR 引入——4 片与 #4599 floor 148 判定的 master 同批红完全重合;另 3 片全是 shard1,本地复算证明新增测试文件触发 LPT 分片重排,born-red 文件被移入 shard1(含 generation-boundary,它在 merge-base 之后才于 master 修好,PR head 上必红)。PR 触碰路径的全部定向套件本地绿(新套件 sqlite lane、storage-unified、store、identity-proof、media-cas、pins-storage-refactor、storage-ext、typecheck)。失败日志 API 取不到(404),按惯例结论已评论到 #4599,未开新 issue。mysql-contract job(本 PR 唯一真实执行 MySQL 车道的决定性门禁)在我写此回复时仍在运行。

判定:代码审核通过,可合——条件:mysql-contract 转绿。另建议合并前 rebase 当前 master(落后 22 个提交,继承红会少很多、shard 结果更好读)。若 mysql-contract 红了再说,届时我会跟进具体失败。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

状态收尾(完整八问审核见上一条回复,不重复):审核已完成并给出判定——可合,条件 = mysql-contract 转绿。

补充一条最新观测:mysql-contract job 截至我最后一次轮询(约 13:00Z)仍处 in_progress,已运行约 50 分钟——明显偏长。一个值得注意的背景:该 job 文件清单里新加入的 tests/storage-backend-mysql-e2e.test.ts 此前从未在任何 CI lane 真实执行过(这正是本 PR 修的 pass-by 问题),所以这可能是它的第一次真跑,卡住/挂起不能排除。失败日志经 API 取不到(404),无法远程诊断;若它转红我会跟进具体失败,若长时间悬挂则建议 owner 在 Actions 页面看一眼该 job 的实时日志。其余各片结论不变:7 个红 test 片均归因为继承红 + 新增测试文件触发的分片重排,PR 触碰路径定向套件本地全绿,typecheck 干净。

@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: zhipuai-lb/glm-5.3)

评审结论:方向正确、数据步骤需返工,暂不建议合并

修复本体(新增 + 列 + UNIQUE(session_id, seq) + stamp-gated 幂等迁移)核过是对的,真实 MySQL 测试也扎实。但三处要改:

  1. 必须 rebase:schema.ts 与 master 的 v8 migration reads the entire storage content into memory at once: large storage (several GB or more) triggers OOM after | v8 迁移把整个存储内容一次性读进内存:大存储(几GB以上)升级 0.1.207 后 OOM,之后每次启动重炸,老会话永久打不开 #4677 流式化改造冲突;当前 head 落后 256 个提交。
  2. 修复步骤是整表载入内存再逐行删——这正是 MySQL storage backend v8 backfill loads the whole table into memory at once and skips dedup | MySQL 存储后端 v8 回填一次性加载全表且不按 msg_id 去重 #4806 报的缺陷本身(MySQL v8 回填整表进内存)。请改成 v8 migration reads the entire storage content into memory at once: large storage (several GB or more) triggers OOM after | v8 迁移把整个存储内容一次性读进内存:大存储(几GB以上)升级 0.1.207 后 OOM,之后每次启动重炸,老会话永久打不开 #4677 之后的按会话分批流式形态,大库上才跑得动。
  3. 多实例窗口没关:迁移判定(“列存在与否”)和删除执行之间没有互斥,两个 bili 进程同时启动会双删。同一会话内 UNIQUE 约束兜底,但请加跨进程锁(或复用统一存储的现有跨进程锁)再执行删除。

改完这三点即可合并,数据安全面没有别的硬伤。

ci-check added 2 commits October 27, 2026 11:22
The sqlite v8 migration backfilled message identity AND collapsed each
(session_id, msg_id, identity) group to its lowest-seq arrival, refunding
the dropped rows' CAS refs. The MySQL upgrade only backfilled — legacy
libraries carrying wireIndex-split generations replayed every duplicate
row forever, and #4390's historian sweep could never remove them (both
rows hit the incoming identity set).

- init() now runs the same collapse inside ONE transaction after the
  probe-first ALTERs: lowest seq kept per group, media + text CAS refs
  refunded with the refcount > 0 floor that mirrors releaseContent.
- The data step is gated on the bili_meta schema_version stamp instead of
  column presence, so a crash between the ALTERs and the backfill re-runs
  idempotently on the next boot instead of stranding rows at identity=''.
  The stamp advances but never downgrades (GREATEST guard).
- Same issue's facet list, same PR: the sqlite migration's bare decRef
  gains the refcount > 0 floor, its redundant second identity pass
  (rawByIdentityKey map + double identityOfStoredContent per row) is
  dropped, and verifyProjections now checks RETIRED rows too on both
  backends (retire is tombstone-only, so their projections stay valid).

Tests: tests/issue4666-migration-dedup.test.ts pins all of the above on
both lanes (sqlite always; mysql under BILI_MYSQL_TEST_DSN in an isolated
throwaway database, incl. a crash-arm that re-runs the upgrade from a
half-migrated state).
tests/storage-backend-mysql-e2e.test.ts documents itself as running in
this job ('the CI mysql-contract job runs both') but was never added to
its file list, so it skipped in every CI lane (no DSN outside the job).
Add it to the command so the SessionStore-on-mysql wiring is actually
exercised.
@ranxianglei
ranxianglei force-pushed the 2026-10-25_mysql-v8-dedup branch from f64f231 to c5fedde Compare October 27, 2026 03:31
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

冲突已解决,分支已 rebase 到当前 master 并强推(auto-fix 车道 attempt 1/2 完成)。新 head:c5feddeee(原 f64f23109)。master 现在是该分支的直接祖先,GitHub 的 mergeable 状态会自动转干净(我查 API 时撞上账号级限流,读不到实时标志,但「master 是 PR head 祖先」这点在 git 侧已验证,冲突在结构上不可能再存在)。无需 design 决策——冲突是纯文本性的,双方意图无矛盾,没有开 decision request。

冲突怎么解的:唯一冲突文件 src/storage/schema.ts,两处冲突区。根因:master 的 #4677 流式化改造把 sqlite v8 回填整个重写成了「按会话分批、每会话一个事务、可断点续跑」的新架构,而本 PR 是在旧整表载入形态上做的去重工作,两边改了同一段。解法取意图并集:保留 master 的流式架构不动,把本 PR 的两个语义增量重新落上去——① decRef 加 refcount > 0 下限(防损坏账本迁移中途变负,PR 披露表里声明过的 facet);② 「删冗余第二遍 identity 计算」这一 facet 被 master 的重写结构性满足(流式代码每行只算一次 identity),无需额外动作。其余 4 个文件(ci.yml / mysql.ts / store.ts / 新测试)git 自动合并成功,我逐块核对过:所有 #4666 语义标记与代码块完整无损,PR 相对新 base 的改动仍是原来那 5 个文件、范围不变(schema.ts 从 26 行降到 6 行是因为流式重写吸收了其中一面,不是丢了内容)。

本地验证(定向,非全量):typecheck 干净;master 自己的 tests/issue4677-v8-migration-streaming.test.ts 6/6 绿(证明我合入的下限没破坏流式路径);本 PR 的 tests/issue4666-migration-dedup.test.ts sqlite 车道全绿(mysql 车道需 DSN,本地 skip,由 mysql-contract job 把关)。全量判定以推送后远端 CI 为准。

结构化八问

  1. 是什么问题:PR 分支落后 master 261 个提交且处于 dirty(冲突)状态,无法合并——这是合并性问题,不是代码缺陷。
  2. 什么原因导致的:v8 migration reads the entire storage content into memory at once: large storage (several GB or more) triggers OOM after | v8 迁移把整个存储内容一次性读进内存:大存储(几GB以上)升级 0.1.207 后 OOM,之后每次启动重炸,老会话永久打不开 #4677 流式化与本 PR 的去重改造各自独立重写了 schema.ts 的同一段 v8 迁移代码,文本冲突;意图层面两者兼容(一个是更好的架构,一个是安全 facet)。
  3. 影响范围与严重程度:仅本 PR 的合并状态;对用户零影响;严重度 = PR 阻塞,不解决就一直合不了。
  4. 怎么修复的:rebase 到当前 master;冲突按意图并集解决(master 流式架构 + PR 的 refcount 下限);其余文件自动合并后逐块审计确认无内容丢失;--force-with-lease 钉住原 head 强推,防止覆盖并发推送。
  5. 是否有回归问题:逐项审过——master 流式迁移语义除下限外零变化(下限只在账本已损坏时生效,严格更安全,属 PR 自己披露的行为);PR 五个文件的原内容全部保留;无阈值/时序/输出格式漂移。master 自带的 issue4677 套件合并后本地全绿。
  6. 是否破坏缓存:不涉及(存储层迁移);CAS 引用计数处理保持原样,下限只会阻止负数不会多减。
  7. 是否改了设计方向:无转向——本次解决反而让 PR 对齐了 master 新的流式方向(评审第 2 条要求 MySQL 侧也改流式,正是同一方向)。
  8. 是否引入新配置:无。

分工说明:评审(floor 10)第 2 条(MySQL 去重步骤改按会话流式)和第 3 条(跨进程锁)是代码改造,已转 ci-worker-r1 处理,本次 rebase 刻意没有代做;他们的提交将落在这个 rebase 后的基线上。若其推送与本分支产生新的文本冲突,系统会再次自动转发。

判定:冲突解决完毕,分支对当前 master 干净可合;剩余门禁 = 新 head 的远端 CI(含 mysql-contract)+ ci-worker-r1 的两项改造落地。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

merge-guard 本轮结果:无需推送——冲突已由先前的 rebase 解决(新 head c5feddeee,base e4cc68575 = 当前 master tip,master 已是 PR head 的直接祖先,git 侧不可能再有冲突);GitHub API 确认 mergeable=true,mergeable_state=blocked(仅因 CI 未跑完,不再是 dirty/conflict)。该 head 的 check runs:21 success、11 skipped、14 queued(in_progress),含 mysql-contract,暂无失败。无设计决策点,不开 decision request。

ci-check added 2 commits October 27, 2026 12:24
…ck (#4666 review)

Rework per PR #4678 review: the v8 data step no longer materializes the
whole messages x content join in memory before deleting rows. It now
mirrors the sqlite #4677 streaming design:

- one session per transaction, 500-row seq-cursor pages; memory holds one
  page window plus one session's keeper keys, never the stored bytes
  (closes the OOM class reported in #4806);
- dedup happens inside each session's ordered scan (first arrival kept),
  so duplicates whose two generations sit on different pages are still
  paired;
- a crash between sessions leaves done sessions committed; the stamp gate
  resumes on next boot without reprocessing them;
- rows whose content bytes are missing from the CAS get the printable
  absent marker (byte-parity with schema.ts) instead of rescanning forever;
- progress log every 100 sessions plus start/complete/failure guidance.

Concurrency (review item 3): the data step runs under a server-side named
lock GET_LOCK('bili_v8_upgrade_<db>'). Two bili processes booting against
the same library can no longer double-delete dup groups or double-refund
CAS refs. The lock is connection-scoped (auto-released if the migrator
dies), scoped per database, and waiters retry until acquired — the
migration is resumable, so the latecomer finds nothing left.

Tests (tests/issue4666-migration-dedup.test.ts): harness extracted for the
mysql scenarios; new pins — cross-page-boundary dedup (503-row session)
and concurrent-boot serialization (a second connection holds the lock;
data planted behind it must still be processed, proving enumeration
happens after acquire); crash arm gains a byte-exact absent-marker pin.
…runs once (#4666)

meta_value is VARCHAR(256), which the mysql2 driver returns as a JS string.
The gate in init() read it through the number-only num() helper, so it always
saw 0 and re-entered the v8 data step on every process boot — a full
SELECT DISTINCT ... WHERE identity='' scan plus a migration-lock cycle per
startup instead of exactly once. On a multi-GB library that recurring scan
partially undercuts the #4806 streaming fix. Parse the decimal string
explicitly (Number + isFinite guard); the stored value is unchanged, only the
read-back coercion is corrected, so the stamp now actually gates the step.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

处理结果:评审三点已落实,另修一处未被点名的隐患

先交代一个过程事实:接手时 PR 分支上已经有一版完整的「流式 + 加锁」实现(6937a1dbd,另一条自动化车道先行推上去的)。我本也独立写了一版,逐点核对后发现那版更完整(锁名按库隔离、拿不到锁就阻塞等待而非跳过本轮),为避免覆盖并行成果,我没有推我那版,而是把它当作基线逐点核验,并在其上补了一个它遗留的缺陷(见第 4 问 / 下方「本轮新增修复」)。

对评审三点的核对结论:

  1. rebase — 分支 merge-base = 当前 master tip,已无落后。✅
  2. 整表载入内存 — 已改按会话分批流式:每会话一个事务、500 行 seq 游标分页、LEFT JOIN content,内存只持一页窗口 + 单会话 keeper keys,不再把整表 join 物化进堆(对齐 sqlite v8 migration reads the entire storage content into memory at once: large storage (several GB or more) triggers OOM after | v8 迁移把整个存储内容一次性读进内存:大存储(几GB以上)升级 0.1.207 后 OOM,之后每次启动重炸,老会话永久打不开 #4677 形态);崩溃后已提交会话不重跑。✅ 关掉了 MySQL storage backend v8 backfill loads the whole table into memory at once and skips dedup | MySQL 存储后端 v8 回填一次性加载全表且不按 msg_id 去重 #4806 的 OOM 面。
  3. 多实例窗口 — 数据步骤跑在服务端命名锁(GET_LOCK,锁名按当前库 DATABASE() 隔离)下:连接级(进程死则自动释放)、拿不到就重试直到拿到(等对方跑完再进,此时已无可删);锁与全部迁移查询走同一连接。✅

本轮新增修复(625a68b5f,评审未点名但确实存在):
init() 的迁移门读 meta_value(VARCHAR(256),驱动按字符串返回)时走了只认 number 的 num() helper,恒得 0 → 0 < 8 永远成立 → 每次进程启动都重进 v8 数据步骤,白做一次全表 SELECT DISTINCT … WHERE identity='' 扫描 + 一轮取锁。幂等 + 有锁所以不坏数据,但在 GB 级库上是每次启动的重复全扫,部分抵消 #4806 收益,也让「以 stamp 为门」的设计形同虚设。改为显式解析十进制串(Number + isFinite 兜底),存储值不变、只纠正读回的类型强转,stamp 现在真正起到「只跑一次」的门控。

结构化八问

  1. 是什么问题:MySQL v7→v8 升级把 wireIndex 拆分留下的重复消息行永久回放(Duplicate message rows not cleaned up after upgrading legacy MySQL to v8 (the SQLite version did clean them): migration | MySQL 老库升 v8 后重复消息行没被清理(sqlite 版清了):迁移少一步去重,回放重复且永不自愈 #4666);原始修法另有两处衍生缺陷——回填整表进内存(GB 库 OOM,MySQL storage backend v8 backfill loads the whole table into memory at once and skips dedup | MySQL 存储后端 v8 回填一次性加载全表且不按 msg_id 去重 #4806)、判定与删除之间无跨进程互斥(双实例同时启动会双删/双减 CAS 引用)。
  2. 什么原因导致的:①原 dedup 把整个 messages×content join 物化进 Node 堆;②「该不该跑迁移」的判断与删除执行不在同一互斥域内;③(本轮新修)schema_version 存成 VARCHAR、读回是字符串,却用只认数字的 helper 去比较,门控恒真。
  3. 影响范围与严重程度:所有走 MySQL 后端、从 v7 升到 v8 的实例。升前:大库首开可能 OOM(MySQL storage backend v8 backfill loads the whole table into memory at once and skips dedup | MySQL 存储后端 v8 回填一次性加载全表且不按 msg_id 去重 #4806)、罕见并发下双删——属功能/可用性缺陷,非数据丢失。本轮修的 stamp 门控:每次启动多一次空扫,性能瑕疵(幂等有锁、不丢数据)。
  4. 怎么修复的:数据步骤改按会话流式(一页一窗、每会话一事务、可断点续跑)+服务端命名锁串行化跨进程;并修正 stamp 读回强转使其真正只跑一次。
  5. 是否有回归问题:逐项过既有行为——wire/消息形状不变;CAS 引用回收仍「先删引用行再删主行」+refcount>0 兜底(对齐 releaseContent,≤0 交 gc);stamp 用 GREATEST 只升不降;verifyProjections 现纳入 retired 行(retired 行损坏时 ok:true→ok:false,属预期收紧)。未见无意回归。
  6. 是否破坏缓存:不涉及缓存失效策略变更;内容 CAS 引用计数回收路径与既有 releaseContent 一致。
  7. 是否改了设计方向:无新架构——把 MySQL 升级路径对齐到既有 sqlite v8 migration reads the entire storage content into memory at once: large storage (several GB or more) triggers OOM after | v8 迁移把整个存储内容一次性读进内存:大存储(几GB以上)升级 0.1.207 后 OOM,之后每次启动重炸,老会话永久打不开 #4677 流式设计(两后端 parity),并补上跨进程锁这一缺失维度。
  8. 是否引入了新的配置:无新增 env/config/CLI。V8_BACKFILL_PAGE、锁名、轮询秒数均为模块内部常量,不进配置面。

验证边界(诚实声明):本地无 MySQL,mysql lane 无法本地跑,上述代码路径靠逐行核对 + typecheck(绿)确认;sqlite lane 本地已跑通。全量与真实 MySQL 的判定以 push 后 CI 的 mysql-contract job 为准——红了再定向排查。

收工判定:评审三点已全部落在分支并经我独立核对,另修一处 stamp 门控隐患;数据安全面无硬伤。CI mysql-contract 绿即可合并。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

merge-guard 收工(本会话两次唤醒,结论一致):均为早于先前 rebase 推送的过期/重复派单——git 侧 master(e4cc68575)已是 PR head c5feddeee 的直接祖先(behind=0),GitHub API mergeable=true、mergeable_state=blocked(仅 CI 未跑完,非 dirty/conflict)。未做任何推送;详细状态见本线程我上一条结果评论。后续若 ci-worker-r1 的流式化+跨进程锁改造推送后出现真实新冲突,再按冲突流程处理。

…view round 2)

storage-backend-mysql-e2e.test.ts claimed in its header that the CI
mysql-contract job runs it, but no workflow ever listed it — this PR's
chore commit gave it its first real execution (2026-10-27), which failed
and then hung the job ~10 minutes until external cancellation:

- a failed assertion meant neither store ever reached closeQuietlyAsync;
  the open mysql pool kept the event loop alive on its sockets, so the
  runner could never exit. finally now closes both stores (allSettled,
  so the success-path double close cannot mask the original failure);
- untilRows' default budget 5s -> 30s (CI docker-mysql latency margin
  under a loaded 2-vCPU runner);
- the mysql-contract job pins --test-concurrency=1: all four listed
  files share ONE DSN/database, and node's default file parallelism
  (cores-1) would let two suites race on the same tables once runners
  grow past 2 vCPUs.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

这条 CI 失败是陈旧的(stale),不是当前代码的问题——无需修复,我没有做任何改动。

自动修复车道引用的 run 40186317614 跑在 Oct 25 的旧 head f64f231095 上——那是 rebase + 流式化改造之前一个已被取代的提交,现在分支上早已不存在。它报的那 7 个 test 分片红,反映的是那个死掉的旧状态。

当前真实状态(刚查 GitHub API 核实):

  • master e4cc68575 是绿的:其 CI run 40439327344 completed success(今天 02:35Z),Release / Release Kernel / Build Artifact 也全绿 → 没有继承性红,不需要 merge ci-fixes。
  • 本 PR 当前 head 625a68b5f:新一轮 run 40450641796 里 27 个 job 已有 25 个 success——包括全部通用 test 分片 + 全部 e2e 车道 + Build Artifact。唯一还在跑的是 mysql-contract(04:37Z 起跑),另有一个条件门 red-signal 正常 skip。当前代码零失败。
  • 通知里那 7 个红的 test 分片,在当前 head 上全是绿的。

为什么本轮改动不可能造成通用分片红:diff 只动 DSN-gated 的 MySQL 升级路径(没有 BILI_MYSQL_TEST_DSN 时整段跳过),通用 test 分片根本执行不到;真正跑这段代码的只有 mysql-contract job(带真实 MySQL service container,含流式跨页 + 并发启动两个回归测试)。

处置:没有可修的东西,未做投机性改动(去改一份当前全绿的代码只会引入风险)。等 mysql-contract 收尾出最终结论——它绿即收工;万一它真红,我再按失败签名定位。按 2026-10-24 政策,陈旧产物不立新 issue、不开无谓 fix PR。

结构化八问:

  1. 是什么问题:自动车道收到一条 CI 失败告警,但它指向的是两天前一个已被取代的旧提交(f64f231095),不是当前代码。
  2. 什么原因导致的:分支在 Oct 25→27 间经历了 rebase + 流式化重做,旧 head 被新提交链取代;告警系统仍在引用那个旧 run/SHA。
  3. 影响范围与严重程度:对当前代码零影响——当前 head 的全部 test 分片与 e2e 都绿,master 也绿。严重度:无(误报/陈旧)。
  4. 怎么处理的:查证而非盲改——比对 run 时间戳与 SHA、确认 master 绿、逐一核对当前 head 各分片状态、确认 diff 只触 DSN-gated 路径;据此判定为陈旧告警,不动代码。
  5. 是否有回归问题:否。本轮未改动任何代码,无新增回归面。
  6. 是否破坏缓存:不涉及(无代码变更)。
  7. 是否改了设计方向:否。
  8. 是否引入新配置:否。

结论:无需合并任何修复;mysql-contract 最终结果出来前保持现状,绿则收工、红则再定位。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

attempt 2/2:mysql-contract 仍红,但根因已钉死,且不是 #4666 修复的问题——按车道政策升级为 p1 决策请求,停手等拍板。

核对结论(拉了 job 日志)

为何停手而非直接改:本环境无本地 MySQL,任何对该子系统的改动都无法本地验证(盲改即未验证提交);且处置是「回滚他人刻意加的 CI 覆盖 vs 排查陌生子系统」的取舍,属 owner 拍板范畴。已开 decision request #4925(p1),列了 A/B/C 与推荐(A:先把该 e2e 套件移出 #4678 这个 job、单独立项跟踪其 CI 就绪,让全绿的核心修复先收工)。我没有擅自 revert/gate,避免覆盖 c5feddeee 的刻意决定。

八问

  1. 是什么问题:CI mysql-contract 红在一条 e2e 用例——首次保存的会话行 30s 内没落到 MySQL。
  2. 什么原因导致的:该套件首次进 CI,暴露其异步 ingest→MySQL 落库路径的确定性潜在缺陷,与 Duplicate message rows not cleaned up after upgrading legacy MySQL to v8 (the SQLite version did clean them): migration | MySQL 老库升 v8 后重复消息行没被清理(sqlite 版清了):迁移少一步去重,回放重复且永不自愈 #4666 去重修复无关。
  3. 影响范围与严重程度:仅本 PR 一个 job 一条用例;对 Duplicate message rows not cleaned up after upgrading legacy MySQL to v8 (the SQLite version did clean them): migration | MySQL 老库升 v8 后重复消息行没被清理(sqlite 版清了):迁移少一步去重,回放重复且永不自愈 #4666/MySQL storage backend v8 backfill loads the whole table into memory at once and skips dedup | MySQL 存储后端 v8 回填一次性加载全表且不按 msg_id 去重 #4806 实际功能零影响(专属测试全绿);严重度=CI 阻塞,需决策。
  4. 怎么处理:拉日志定位到具体断言与路径,确认与我改动无关;因无法本地验证 + 属取舍,升级 p1 决策请求 decision request: MySQL e2e test fails its first CI run — defer it from PR #4678 or debug the async-ingest path? #4925。
  5. 是否有回归问题:无——挂死已修,job 能跑完出汇总;我的改动未变且测试通过。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:否。
  8. 是否引入新配置:否(选项 A 若采纳会加一个 env opt-in 门,属 CI 参数,待拍板后由后续处理)。

结论:按 attempt 2/2 政策停止改动 PR,等 owner 在 #4925 拍板 A/B;#4678 的核心修复本身完好且测试全绿。

…th failure is fixed (#4926)

The suite's first two real CI runs (heads 6937a1d / a476a90 of PR
#4678) failed deterministically in the same test — "mysql hot path: saves
land on the backend ...": after writeNow, shadow.viewForAsync never sees
the session row within a full 30s budget, with ZERO log lines in between
(the async record path only logs after 3 consecutive failures, so a
single silent ingest failure is invisible). The failing code paths are
byte-identical to master (at that point the shared test DB was already
stamped v8, so this PR's init() was a no-op there); the suite has never
run in CI before this PR listed it, i.e. it rotted unverified since
2026-10-20. Pre-existing defect, not caused by this PR's diff.

- ci.yml: drop tests/storage-backend-mysql-e2e.test.ts from the
  mysql-contract job list (keep issue4666-migration-dedup.test.ts —
  5/5 green twice against real MySQL 8.0); comment pins the tracking
  issue so a future cleanup doesn't blindly re-add the file.
- untilRows now surfaces shadow.stats.lastError in its timeout message:
  the next red for this suite self-diagnoses instead of going blind
  again (#4926 step 1).

Re-adding the file to the job list happens when #4926 lands.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

状态:42aad38c3 已推——把 tests/storage-backend-mysql-e2e.test.ts 从 mysql-contract job 列表摘除(其 hot-path 测试在真 MySQL 上确定性失败,属既有缺陷、与本 PR diff 无关,取证见 #4666 最新回复;跟踪于新 issue #4926,转绿后按该 issue 加回)+ untilRows 超时信息自诊断化。同时吸收了 ci-worker 的 625a68b5f(schema_version 戳解析:v8 数据步从「每次启动重跑」改为「每库一次」)。等 42aad38 的 CI,其余各项已全绿。

@ranxianglei
ranxianglei merged commit 7926c94 into master Oct 27, 2026
47 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.880-master.7926c94 (rolling master tag).

npm install -g billion-context@0.0.880-master.7926c94  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

ranxianglei pushed a commit that referenced this pull request Oct 27, 2026
Follow-up merge after master advanced past c499b14 mid-resolution (#4969/#4970/#4972/#4971 blog+billing, #4544 event-loop-yield, #4678 mysql-v8-dedup, #4782 dsh-goal-subfork). Conflicts:
- src/server/prepare-responses.ts: import hunk unioned again - kept this branch's fold-reconcile import line (superset of master's: adds noteBiliToolSurface/outboundBiliSurface); master's yieldToEventLoop lines from #4528 landed in all four prepare files untouched.
- tests/golden/storage-render/{anthropic,chat,google,responses}.json: regenerated from the merged tree via scripts/update-storage-render-goldens.ts; output byte-identical to the previous round's regeneration (master's new commits did not change these golden conversations' outbound wire or persisted state) and differing from master only by this branch's image-rescale armed-state latch digests.
Verification: no conflict markers remain; targeted suites green (golden-storage-render + fold-reconcile + issue3400-image-rescale = 109/109); typecheck clean. Full-suite verdict deferred to CI on the pushed head.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant