Repository navigation
[PR] fix: MySQL v8 upgrade cleans up duplicate message rows left by wireIndex splits (#4666) - #4678
Conversation
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.6401-pr.4678.fc0cfb4This exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr4678.tgz
npm install -g packageInstall instructions are refreshed on each push. |
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) 查重: 不重复。 #4678 是 #4666(MySQL v8 迁移缺去重)的唯一在途 PR;open 侧无同症状/同机制条目。相邻的 #4672(PR for #4665,archive/restore 的 identity+retired_at 携带)是不同失败面(#4390 身份清扫 vs 本 PR 的 v8 迁移去重),互不覆盖。 状态:agent 已完工 PR(首行 ai 自声明),我按「识别自己的 PR」流程处理——不重做,拉分支核对代码 + 确认 CI。当前 CI:静态门禁全绿(kernel-guard / config-surface-guard / lane-guard / version-guard 等已完成 success),test shards + |
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) #4678 独立审核完成(未照抄 issue 结论,diff 逐段重读、机制逐条重验)。查重已在首条回复给出:不重复。 审核做了什么:完整读了 5 个文件的 diff 与相关既有代码(identity.ts、store.ts retire/sweep、mysql.ts tx/getMessages、schema.ts migrateSchema、CI job 定义、新测试套件全文);验证了崩溃矩阵、幂等性、调用者面;本地在 PR head worktree 跑了新套件(sqlite lane)+ 6 个与 diff 直接耦合的存储套件 + typecheck;CI 红片做了分片复算归因。 结构化八问
测试质量:断言有牙——缺去重则 survivors=8≠4、缺退还则 refcount=1≠0、缺下限则 -1≠0,三条都能抓住原 bug;crash arm 从半迁移态(戳 '7'、列已存在、identity='' 行)重启,直接区分「戳门控」与「列存在门控」;retired-row 校验先钉干净通过再钉损坏失败,双向都有。小瑕疵(不阻塞):swap 分支( CI 状态:7 个红 test 片经核对均非本 PR 引入——4 片与 #4599 floor 148 判定的 master 同批红完全重合;另 3 片全是 shard1,本地复算证明新增测试文件触发 LPT 分片重排,born-red 文件被移入 shard1(含 generation-boundary,它在 merge-base 之后才于 master 修好,PR head 上必红)。PR 触碰路径的全部定向套件本地绿(新套件 sqlite lane、storage-unified、store、identity-proof、media-cas、pins-storage-refactor、storage-ext、typecheck)。失败日志 API 取不到(404),按惯例结论已评论到 #4599,未开新 issue。 判定:代码审核通过,可合——条件: |
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) 状态收尾(完整八问审核见上一条回复,不重复):审核已完成并给出判定——可合,条件 = 补充一条最新观测: |
|
(ai: zhipuai-lb/glm-5.3) 评审结论:方向正确、数据步骤需返工,暂不建议合并修复本体(新增 + 列 + UNIQUE(session_id, seq) + stamp-gated 幂等迁移)核过是对的,真实 MySQL 测试也扎实。但三处要改:
改完这三点即可合并,数据安全面没有别的硬伤。 |
The sqlite v8 migration backfilled message identity AND collapsed each (session_id, msg_id, identity) group to its lowest-seq arrival, refunding the dropped rows' CAS refs. The MySQL upgrade only backfilled — legacy libraries carrying wireIndex-split generations replayed every duplicate row forever, and #4390's historian sweep could never remove them (both rows hit the incoming identity set). - init() now runs the same collapse inside ONE transaction after the probe-first ALTERs: lowest seq kept per group, media + text CAS refs refunded with the refcount > 0 floor that mirrors releaseContent. - The data step is gated on the bili_meta schema_version stamp instead of column presence, so a crash between the ALTERs and the backfill re-runs idempotently on the next boot instead of stranding rows at identity=''. The stamp advances but never downgrades (GREATEST guard). - Same issue's facet list, same PR: the sqlite migration's bare decRef gains the refcount > 0 floor, its redundant second identity pass (rawByIdentityKey map + double identityOfStoredContent per row) is dropped, and verifyProjections now checks RETIRED rows too on both backends (retire is tombstone-only, so their projections stay valid). Tests: tests/issue4666-migration-dedup.test.ts pins all of the above on both lanes (sqlite always; mysql under BILI_MYSQL_TEST_DSN in an isolated throwaway database, incl. a crash-arm that re-runs the upgrade from a half-migrated state).
tests/storage-backend-mysql-e2e.test.ts documents itself as running in
this job ('the CI mysql-contract job runs both') but was never added to
its file list, so it skipped in every CI lane (no DSN outside the job).
Add it to the command so the SessionStore-on-mysql wiring is actually
exercised.
f64f231 to
c5fedde
Compare
(ai: vllm-qwen/qwen3.8-27b) 冲突已解决,分支已 rebase 到当前 master 并强推(auto-fix 车道 attempt 1/2 完成)。新 head: 冲突怎么解的:唯一冲突文件 本地验证(定向,非全量):typecheck 干净;master 自己的 结构化八问
分工说明:评审(floor 10)第 2 条(MySQL 去重步骤改按会话流式)和第 3 条(跨进程锁)是代码改造,已转 ci-worker-r1 处理,本次 rebase 刻意没有代做;他们的提交将落在这个 rebase 后的基线上。若其推送与本分支产生新的文本冲突,系统会再次自动转发。 判定:冲突解决完毕,分支对当前 master 干净可合;剩余门禁 = 新 head 的远端 CI(含 |
(ai: vllm-qwen/qwen3.8-27b) merge-guard 本轮结果:无需推送——冲突已由先前的 rebase 解决(新 head |
…ck (#4666 review) Rework per PR #4678 review: the v8 data step no longer materializes the whole messages x content join in memory before deleting rows. It now mirrors the sqlite #4677 streaming design: - one session per transaction, 500-row seq-cursor pages; memory holds one page window plus one session's keeper keys, never the stored bytes (closes the OOM class reported in #4806); - dedup happens inside each session's ordered scan (first arrival kept), so duplicates whose two generations sit on different pages are still paired; - a crash between sessions leaves done sessions committed; the stamp gate resumes on next boot without reprocessing them; - rows whose content bytes are missing from the CAS get the printable absent marker (byte-parity with schema.ts) instead of rescanning forever; - progress log every 100 sessions plus start/complete/failure guidance. Concurrency (review item 3): the data step runs under a server-side named lock GET_LOCK('bili_v8_upgrade_<db>'). Two bili processes booting against the same library can no longer double-delete dup groups or double-refund CAS refs. The lock is connection-scoped (auto-released if the migrator dies), scoped per database, and waiters retry until acquired — the migration is resumable, so the latecomer finds nothing left. Tests (tests/issue4666-migration-dedup.test.ts): harness extracted for the mysql scenarios; new pins — cross-page-boundary dedup (503-row session) and concurrent-boot serialization (a second connection holds the lock; data planted behind it must still be processed, proving enumeration happens after acquire); crash arm gains a byte-exact absent-marker pin.
…runs once (#4666) meta_value is VARCHAR(256), which the mysql2 driver returns as a JS string. The gate in init() read it through the number-only num() helper, so it always saw 0 and re-entered the v8 data step on every process boot — a full SELECT DISTINCT ... WHERE identity='' scan plus a migration-lock cycle per startup instead of exactly once. On a multi-GB library that recurring scan partially undercuts the #4806 streaming fix. Parse the decimal string explicitly (Number + isFinite guard); the stored value is unchanged, only the read-back coercion is corrected, so the stamp now actually gates the step.
(ai: vllm-qwen/qwen3.8-27b) 处理结果:评审三点已落实,另修一处未被点名的隐患先交代一个过程事实:接手时 PR 分支上已经有一版完整的「流式 + 加锁」实现( 对评审三点的核对结论:
本轮新增修复( 结构化八问
验证边界(诚实声明):本地无 MySQL,mysql lane 无法本地跑,上述代码路径靠逐行核对 + typecheck(绿)确认;sqlite lane 本地已跑通。全量与真实 MySQL 的判定以 push 后 CI 的 收工判定:评审三点已全部落在分支并经我独立核对,另修一处 stamp 门控隐患;数据安全面无硬伤。CI |
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) merge-guard 收工(本会话两次唤醒,结论一致):均为早于先前 rebase 推送的过期/重复派单——git 侧 master( |
…view round 2) storage-backend-mysql-e2e.test.ts claimed in its header that the CI mysql-contract job runs it, but no workflow ever listed it — this PR's chore commit gave it its first real execution (2026-10-27), which failed and then hung the job ~10 minutes until external cancellation: - a failed assertion meant neither store ever reached closeQuietlyAsync; the open mysql pool kept the event loop alive on its sockets, so the runner could never exit. finally now closes both stores (allSettled, so the success-path double close cannot mask the original failure); - untilRows' default budget 5s -> 30s (CI docker-mysql latency margin under a loaded 2-vCPU runner); - the mysql-contract job pins --test-concurrency=1: all four listed files share ONE DSN/database, and node's default file parallelism (cores-1) would let two suites race on the same tables once runners grow past 2 vCPUs.
(ai: vllm-qwen/qwen3.8-27b) 这条 CI 失败是陈旧的(stale),不是当前代码的问题——无需修复,我没有做任何改动。 自动修复车道引用的 run 当前真实状态(刚查 GitHub API 核实):
为什么本轮改动不可能造成通用分片红:diff 只动 DSN-gated 的 MySQL 升级路径(没有 处置:没有可修的东西,未做投机性改动(去改一份当前全绿的代码只会引入风险)。等 结构化八问:
结论:无需合并任何修复; |
(ai: vllm-qwen/qwen3.8-27b) attempt 2/2: 核对结论(拉了 job 日志)
为何停手而非直接改:本环境无本地 MySQL,任何对该子系统的改动都无法本地验证(盲改即未验证提交);且处置是「回滚他人刻意加的 CI 覆盖 vs 排查陌生子系统」的取舍,属 owner 拍板范畴。已开 decision request #4925(p1),列了 A/B/C 与推荐(A:先把该 e2e 套件移出 #4678 这个 job、单独立项跟踪其 CI 就绪,让全绿的核心修复先收工)。我没有擅自 revert/gate,避免覆盖 八问
结论:按 attempt 2/2 政策停止改动 PR,等 owner 在 #4925 拍板 A/B;#4678 的核心修复本身完好且测试全绿。 |
…th failure is fixed (#4926) The suite's first two real CI runs (heads 6937a1d / a476a90 of PR #4678) failed deterministically in the same test — "mysql hot path: saves land on the backend ...": after writeNow, shadow.viewForAsync never sees the session row within a full 30s budget, with ZERO log lines in between (the async record path only logs after 3 consecutive failures, so a single silent ingest failure is invisible). The failing code paths are byte-identical to master (at that point the shared test DB was already stamped v8, so this PR's init() was a no-op there); the suite has never run in CI before this PR listed it, i.e. it rotted unverified since 2026-10-20. Pre-existing defect, not caused by this PR's diff. - ci.yml: drop tests/storage-backend-mysql-e2e.test.ts from the mysql-contract job list (keep issue4666-migration-dedup.test.ts — 5/5 green twice against real MySQL 8.0); comment pins the tracking issue so a future cleanup doesn't blindly re-add the file. - untilRows now surfaces shadow.stats.lastError in its timeout message: the next red for this suite self-diagnoses instead of going blind again (#4926 step 1). Re-adding the file to the job list happens when #4926 lands.
(ai: vllm-qwen/qwen3.8-27b) 状态: |
📦 Merged Build PublishedThis merge to master was published as npm install -g billion-context@0.0.880-master.7926c94 # this exact build
npm install -g billion-context@master # always the newest merged build
|
Follow-up merge after master advanced past c499b14 mid-resolution (#4969/#4970/#4972/#4971 blog+billing, #4544 event-loop-yield, #4678 mysql-v8-dedup, #4782 dsh-goal-subfork). Conflicts: - src/server/prepare-responses.ts: import hunk unioned again - kept this branch's fold-reconcile import line (superset of master's: adds noteBiliToolSurface/outboundBiliSurface); master's yieldToEventLoop lines from #4528 landed in all four prepare files untouched. - tests/golden/storage-render/{anthropic,chat,google,responses}.json: regenerated from the merged tree via scripts/update-storage-render-goldens.ts; output byte-identical to the previous round's regeneration (master's new commits did not change these golden conversations' outbound wire or persisted state) and differing from master only by this branch's image-rescale armed-state latch digests. Verification: no conflict markers remain; targeted suites green (golden-storage-render + fold-reconcile + issue3400-image-rescale = 109/109); typecheck clean. Full-suite verdict deferred to CI on the pushed head.
Fixes #4666 · family: #4806 (MySQL OOM facet), template: #4677/#4684 (sqlite streaming)
What this PR does
Legacy MySQL libraries upgraded to v8 keep their pre-#4390 wireIndex-split duplicates forever: several rows per logical message (same msg_id, different content_hash, identical identity once id/wireIndex are stripped). The sqlite v8 migration collapses each (session_id, msg_id, identity) group to its lowest seq and refunds the dropped rows' CAS refs; the MySQL migration backfilled identity but skipped the collapse — affected libraries replay every duplicate forever, and #4390's historian sweep can never remove them (both rows "hit" the incoming identity set).
This PR gives the MySQL upgrade the same collapse step, built as the sqlite lane's streaming design (#4677) plus a cross-process lock:
migrateV8Identities, src/storage/mysql.ts): sessions selected byidentity = '', ONE SESSION PER TRANSACTION over 500-row seq-cursor pages; dedup happens inside each session's ordered scan (first arrival kept, later generations deleted with media + text CAS refs refunded,refcount > 0floor); rows whose content bytes are missing from the CAS get the printableabsent:marker (byte-parity with schema.ts); progress log every 100 sessions. Memory holds one page window + one session's keeper keys — never the whole table's bytes (MySQL storage backend v8 backfill loads the whole table into memory at once and skips dedup | MySQL 存储后端 v8 回填一次性加载全表且不按 msg_id 去重 #4806 class).GET_LOCK('bili_v8_upgrade_<db>')around the data step — connection-scoped (auto-released if the migrator dies), per-database scoped, waiters retry until acquired. Two instances booting against the same library can no longer double-delete dup groups / double-refund refs.refcount > 0floor; the redundant second identity pass is gone; verifyProjections checks RETIRED rows on both backends (diagnostic coverage gap).Review rework (owner review of this PR, 2026-10-27)
Behavior changes (old → new)
absent:marker, parity with sqliterefcount > 0No config-surface change. No kernel/ change. No version bump.
Tests
tests/issue4666-migration-dedup.test.ts (dual-lane; mysql lane runs in the CI mysql-contract job against throwaway databases):