Skip to content

[PR] storage: MySQL backend — step 2 of the pluggable-storage plan (#3931) - #3946

Merged
ranxianglei merged 1 commit into
masterfrom
2026-10-20_mysql-backend
Oct 20, 2026
Merged

ranxianglei merged 1 commit into
masterfrom
2026-10-20_mysql-backend

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Stacked on #3932 (step 1 — the StorageBackend contract); merge that first.

What

Schema notes

  • Fresh-install DDL at the v7 shape (a MySQL deployment starts at v7; the sqlite migration ladder does not apply). bili_meta carries schema_version (replaces PRAGMA user_version).
  • JSON/bytes columns are LONGTEXT, not MySQL JSON: the JSON type normalizes key order and would break the byte-exact digest contract.
  • key → ext_key / kv_key (reserved words); utf8mb4 + InnoDB; unique key lengths sized under the 3072-byte cap.

Explicitly not here (step 3, tracked in #3931)

The proxy hot path (shadow ingest / persist.loadSync's sync kernel contract) still runs on sqlite — swapping it means the persist async-ification, which is its own PR. The boot notice names this boundary. Until then mysql is fully usable for contract consumers (export today; replicas/checkpoints/gc follow in step 3).

Verification

  • tests/storage-backend-contract.test.ts: 6 sqlite assertions + a full mysql lane (ingest round-trip, ext/watermark, identical re-ingest stability, reap journal, archive→restore→gc, refcounted delete, verifyProjections) — 7/7 green against a real MySQL 8.0.46 (docker run mysql:8.0).
  • storage-backend-knob 4/4 (DSN precedence + async openBackend semantics), export-jsonl 4/4, storage-unified 14/14, archive-gc 5/5, reap-journal 2/2.
  • tsc clean; unused-exports / config-surface / gen-config-docs gates clean; docs seeds updated (storage.mysql.dsn, BILI_MYSQL_DSN).

Fixes #3388 (step 2 of the plan; step 3 — the hot-path swap — stays tracked in #3931).

…e full StorageBackend contract, mysql2 bundled + lazily loaded, export speaks mysql today (#3931)

- src/storage/mysql.ts: full contract surface against MySQL 8.0 (ingest with
  atomic CAS refcount bumps, delta/full/completeLog paths, reads, session_ext,
  reap journal, checkpoints, conversations, archive/restore/gc with the local
  JSONL archive log, diagnostics). DDL is the v7 shape (bili_meta carries the
  schema version; JSON columns stay LONGTEXT for byte-exact digests).
- backend.ts: openBackend is async; the mysql branch lazily imports mysql2
  (boots that never select mysql never load it); resolveMysqlDsn
  (env BILI_MYSQL_DSN > storage.mysql.dsn > documented error).
- export.ts: backend-aware open — storage.backend="mysql" + DSN exports
  straight from MySQL; every fallback stays on the sqlite index.
- Contract suite: the mysql lane runs the same assertions against a real
  mysql:8.0 (BILI_MYSQL_TEST_DSN); CI gains a mysql-contract service lane.
- Boundary (step 3, next PR): the proxy hot path (shadow ingest) stays on
  sqlite until the persist async swap; the boot notice names the boundary.
@ranxianglei
ranxianglei force-pushed the 2026-10-20_mysql-backend branch from 48c0384 to 288c795 Compare October 20, 2026 11:49
@ranxianglei
ranxianglei changed the base branch from 2026-10-20_storage-backend-seam to master October 20, 2026 11:49
@ranxianglei
ranxianglei merged commit 440c511 into master Oct 20, 2026
15 checks passed
ranxianglei pushed a commit that referenced this pull request Oct 20, 2026
… + carry status into mysql summary rows

Rebase onto 1cf118e exposed two semantic collisions: (1) #3931 step 3 renamed UnifiedShadow.store to .sync (null on network backends) — the four CLI maintenance pass-throughs now degrade like their neighboring sync facades (archiveSession returns a reason, stats/gcPlan return null, vacuum throws so bili sessions rm reports the real error); (2) the additive status field on SessionSummaryRow/WebSummaryRow broke MysqlBackend's row literals — both queries now select s.status, mirroring getSessionRow.
ranxianglei pushed a commit that referenced this pull request Oct 20, 2026
… + carry status into mysql summary rows

Rebase onto 1cf118e exposed two semantic collisions: (1) #3931 step 3 renamed UnifiedShadow.store to .sync (null on network backends) — the four CLI maintenance pass-throughs now degrade like their neighboring sync facades (archiveSession returns a reason, stats/gcPlan return null, vacuum throws so bili sessions rm reports the real error); (2) the additive status field on SessionSummaryRow/WebSummaryRow broke MysqlBackend's row literals — both queries now select s.status, mirroring getSessionRow.
ranxianglei pushed a commit that referenced this pull request Oct 20, 2026
… + carry status into mysql summary rows

Rebase onto 1cf118e exposed two semantic collisions: (1) #3931 step 3 renamed UnifiedShadow.store to .sync (null on network backends) — the four CLI maintenance pass-throughs now degrade like their neighboring sync facades (archiveSession returns a reason, stats/gcPlan return null, vacuum throws so bili sessions rm reports the real error); (2) the additive status field on SessionSummaryRow/WebSummaryRow broke MysqlBackend's row literals — both queries now select s.status, mirroring getSessionRow.
Zhishulo pushed a commit to Zhishulo/billion-context that referenced this pull request Oct 20, 2026
…he stranded step-3 merge (ranxianglei#3955)

master is red on two jobs since ranxianglei#3946 merged:
- mysql-contract: package.json lost the mysql2 dependency when a local
  version-restore accident reverted the npm install --save edit (the lock
  kept the subtree), so the service lane cannot resolve mysql2.
- lock-sync: 11 lock entries carried resolved URLs from the local install
  relay (127.0.0.1:4873); rewritten to registry.npmjs.org (integrity
  hashes were already the official ones).

Also folds in 2350797 (the ranxianglei#3955 merge) which was accidentally merged
into its stacked base branch (2026-10-20_mysql-backend) instead of master
— this PR lands step 3 (mysql hot path) on master.
This was referenced Oct 20, 2026
prenom6548 pushed a commit to prenom6548/billion-context that referenced this pull request Oct 21, 2026
…guard (ranxianglei#3260)

Port pi's findDegenerateRuns minRun semantics (200) onto the existing intrinsic
stream terminator: a model stuck in a symbol loop (ranxianglei#3946: one symbol string for
~22 minutes) now trips the stream mid-generation instead of running to
max-output or manual abort.

Detection runs on DECODED model text only — per-frame JSON walk over text-family
keys (openai delta.content/reasoning_content, anthropic text_delta/thinking_delta,
responses top-level delta, google parts[].text). Tool-call arguments are excluded
(ranxianglei#1039 user intent), and Responses cumulative snapshot events (.done/.completed/
.incomplete) are skipped so runs are never double-counted. Every frame is parsed:
token-granular streaming interleaves framing bytes between identical characters, so
a raw run-length pre-gate would stay blind to the incident shape. NDJSON bodies
(bare-newline events) are handled by a fallback splitter; byte-exact forwarding and
the clean-EOF trip contract are unchanged. Always on, zero new config.

Also neutralizes the trip log wording in relay.ts (two sites): it said 'runaway
enumeration detected' even when the reason is now text-run.
prenom6548 pushed a commit to prenom6548/billion-context that referenced this pull request Oct 21, 2026
…ide runaway guard

The fake-model growth filler "x".repeat(360) emits a run of 360 identical
codepoints per reply. Since ranxianglei#3289 merged the output-side runaway guard (ranxianglei#3260,
src/runaway-guard.ts: textRun >= TEXT_RUN_THRESHOLD = 200), every synthetic
reply is aborted mid-stream ("output-side runaway guard tripped (text-run;
maxTextRun:200)") and the client sees an empty response, so T5's "every
client turn must get a visible reply" fails deterministically on every
shard-2 CI lane (run 38777578495: ubu22-s2 / ubu24-s2 / win22-s2 / win24-s2).

Fix at the producer: "x ".repeat(180) — same 360 bytes per reply, max run
of identical codepoints is 1, so the fixture stops emitting the degenerate
symbol-loop shape the guard exists to catch (ranxianglei#3946). Guard untouched; no
runtime code changed.

Verified locally: tests/issue1086-chain-self-exemption.test.ts 21/21.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant