Repository navigation
[PR] storage: MySQL backend — step 2 of the pluggable-storage plan (#3931) - #3946
Merged
Merged
Conversation
ranxianglei
force-pushed
the
2026-10-20_mysql-backend
branch
from
October 20, 2026 11:31
5a606a5 to
48c0384
Compare
ranxianglei
force-pushed
the
2026-10-20_storage-backend-seam
branch
from
October 20, 2026 11:47
5e3a52c to
5d665fa
Compare
…e full StorageBackend contract, mysql2 bundled + lazily loaded, export speaks mysql today (#3931) - src/storage/mysql.ts: full contract surface against MySQL 8.0 (ingest with atomic CAS refcount bumps, delta/full/completeLog paths, reads, session_ext, reap journal, checkpoints, conversations, archive/restore/gc with the local JSONL archive log, diagnostics). DDL is the v7 shape (bili_meta carries the schema version; JSON columns stay LONGTEXT for byte-exact digests). - backend.ts: openBackend is async; the mysql branch lazily imports mysql2 (boots that never select mysql never load it); resolveMysqlDsn (env BILI_MYSQL_DSN > storage.mysql.dsn > documented error). - export.ts: backend-aware open — storage.backend="mysql" + DSN exports straight from MySQL; every fallback stays on the sqlite index. - Contract suite: the mysql lane runs the same assertions against a real mysql:8.0 (BILI_MYSQL_TEST_DSN); CI gains a mysql-contract service lane. - Boundary (step 3, next PR): the proxy hot path (shadow ingest) stays on sqlite until the persist async swap; the boot notice names the boundary.
ranxianglei
force-pushed
the
2026-10-20_mysql-backend
branch
from
October 20, 2026 11:49
48c0384 to
288c795
Compare
ranxianglei
changed the base branch from
2026-10-20_storage-backend-seam
to
master
October 20, 2026 11:49
This was referenced Oct 20, 2026
ranxianglei
pushed a commit
that referenced
this pull request
Oct 20, 2026
… + carry status into mysql summary rows Rebase onto 1cf118e exposed two semantic collisions: (1) #3931 step 3 renamed UnifiedShadow.store to .sync (null on network backends) — the four CLI maintenance pass-throughs now degrade like their neighboring sync facades (archiveSession returns a reason, stats/gcPlan return null, vacuum throws so bili sessions rm reports the real error); (2) the additive status field on SessionSummaryRow/WebSummaryRow broke MysqlBackend's row literals — both queries now select s.status, mirroring getSessionRow.
ranxianglei
pushed a commit
that referenced
this pull request
Oct 20, 2026
… + carry status into mysql summary rows Rebase onto 1cf118e exposed two semantic collisions: (1) #3931 step 3 renamed UnifiedShadow.store to .sync (null on network backends) — the four CLI maintenance pass-throughs now degrade like their neighboring sync facades (archiveSession returns a reason, stats/gcPlan return null, vacuum throws so bili sessions rm reports the real error); (2) the additive status field on SessionSummaryRow/WebSummaryRow broke MysqlBackend's row literals — both queries now select s.status, mirroring getSessionRow.
ranxianglei
pushed a commit
that referenced
this pull request
Oct 20, 2026
… + carry status into mysql summary rows Rebase onto 1cf118e exposed two semantic collisions: (1) #3931 step 3 renamed UnifiedShadow.store to .sync (null on network backends) — the four CLI maintenance pass-throughs now degrade like their neighboring sync facades (archiveSession returns a reason, stats/gcPlan return null, vacuum throws so bili sessions rm reports the real error); (2) the additive status field on SessionSummaryRow/WebSummaryRow broke MysqlBackend's row literals — both queries now select s.status, mirroring getSessionRow.
Zhishulo
pushed a commit
to Zhishulo/billion-context
that referenced
this pull request
Oct 20, 2026
…he stranded step-3 merge (ranxianglei#3955) master is red on two jobs since ranxianglei#3946 merged: - mysql-contract: package.json lost the mysql2 dependency when a local version-restore accident reverted the npm install --save edit (the lock kept the subtree), so the service lane cannot resolve mysql2. - lock-sync: 11 lock entries carried resolved URLs from the local install relay (127.0.0.1:4873); rewritten to registry.npmjs.org (integrity hashes were already the official ones). Also folds in 2350797 (the ranxianglei#3955 merge) which was accidentally merged into its stacked base branch (2026-10-20_mysql-backend) instead of master — this PR lands step 3 (mysql hot path) on master.
This was referenced Oct 20, 2026
Closed
Merged
prenom6548
pushed a commit
to prenom6548/billion-context
that referenced
this pull request
Oct 21, 2026
…guard (ranxianglei#3260) Port pi's findDegenerateRuns minRun semantics (200) onto the existing intrinsic stream terminator: a model stuck in a symbol loop (ranxianglei#3946: one symbol string for ~22 minutes) now trips the stream mid-generation instead of running to max-output or manual abort. Detection runs on DECODED model text only — per-frame JSON walk over text-family keys (openai delta.content/reasoning_content, anthropic text_delta/thinking_delta, responses top-level delta, google parts[].text). Tool-call arguments are excluded (ranxianglei#1039 user intent), and Responses cumulative snapshot events (.done/.completed/ .incomplete) are skipped so runs are never double-counted. Every frame is parsed: token-granular streaming interleaves framing bytes between identical characters, so a raw run-length pre-gate would stay blind to the incident shape. NDJSON bodies (bare-newline events) are handled by a fallback splitter; byte-exact forwarding and the clean-EOF trip contract are unchanged. Always on, zero new config. Also neutralizes the trip log wording in relay.ts (two sites): it said 'runaway enumeration detected' even when the reason is now text-run.
prenom6548
pushed a commit
to prenom6548/billion-context
that referenced
this pull request
Oct 21, 2026
…ide runaway guard The fake-model growth filler "x".repeat(360) emits a run of 360 identical codepoints per reply. Since ranxianglei#3289 merged the output-side runaway guard (ranxianglei#3260, src/runaway-guard.ts: textRun >= TEXT_RUN_THRESHOLD = 200), every synthetic reply is aborted mid-stream ("output-side runaway guard tripped (text-run; maxTextRun:200)") and the client sees an empty response, so T5's "every client turn must get a visible reply" fails deterministically on every shard-2 CI lane (run 38777578495: ubu22-s2 / ubu24-s2 / win22-s2 / win24-s2). Fix at the producer: "x ".repeat(180) — same 360 bytes per reply, max run of identical codepoints is 1, so the fixture stops emitting the degenerate symbol-loop shape the guard exists to catch (ranxianglei#3946). Guard untouched; no runtime code changed. Verified locally: tests/issue1086-chain-self-exemption.test.ts 21/21.
This was referenced Oct 24, 2026
Open
This was referenced Nov 11, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #3932 (step 1 — the StorageBackend contract); merge that first.
What
src/storage/mysql.ts— MysqlBackend implements the full StorageBackend contract against MySQL 8.0: ingest (delta / full / completeLog with atomic CAS refcount bumps —INSERT … ON DUPLICATE KEY UPDATE refcount = refcount + 1), reads (getMessageswith media hydration,getBlocks,getRefs,getDeadRefs,getCcr/getCcrEntries),session_ext+kv, the Reaped blocks resurrect after restart in full mode: stale index row outlives the #2695 reap and poisons the JSON envelope #3841 reap journal, checkpoints (CAS state blobs), conversation bindings, archive/restore/gc (same localarchive/archive.jsonlsingle-file log as sqlite, [PR] storage: GC archive as one append-only JSONL log (follow-up to #3911) #3927), and the diagnostics surface (verifyProjections,counts,contentDedupReport, refcount probes).openBackend("mysql", dsn)dynamic-imports./mysql.js; boots that never select mysql never load the driver (the Storage backend seam (step 1): async operation-level contract + sqlite re-wired through it — MySQL stays a registered stub (#3388) #3931 owner decision — pure JS ~500KB, lockfile pins the version).env BILI_MYSQL_DSN > storage.mysql.dsn > documented error(the error names both knobs and the sqlite default).storage.backend="mysql"+ a DSN and the JSONL export renders straight from MySQL; every failure degrades to the local sqlite index / legacy-tail renderer exactly like a corrupt index does.mysql-contractlane: a mysql:8.0 service container runs the same contract suite on every PR (BILI_MYSQL_TEST_DSN); the sqlite lane stays always-on inside the main test job.Schema notes
bili_metacarriesschema_version(replaces PRAGMA user_version).key→ext_key/kv_key(reserved words); utf8mb4 + InnoDB; unique key lengths sized under the 3072-byte cap.Explicitly not here (step 3, tracked in #3931)
The proxy hot path (shadow ingest /
persist.loadSync's sync kernel contract) still runs on sqlite — swapping it means the persist async-ification, which is its own PR. The boot notice names this boundary. Until then mysql is fully usable for contract consumers (export today; replicas/checkpoints/gc follow in step 3).Verification
tests/storage-backend-contract.test.ts: 6 sqlite assertions + a full mysql lane (ingest round-trip, ext/watermark, identical re-ingest stability, reap journal, archive→restore→gc, refcounted delete, verifyProjections) — 7/7 green against a real MySQL 8.0.46 (docker run mysql:8.0).storage-backend-knob4/4 (DSN precedence + async openBackend semantics), export-jsonl 4/4, storage-unified 14/14, archive-gc 5/5, reap-journal 2/2.storage.mysql.dsn,BILI_MYSQL_DSN).Fixes #3388 (step 2 of the plan; step 3 — the hot-path swap — stays tracked in #3931).