Skip to content

[PR] fix(dist): load mysql2 lazily at backend-open time + generalize the dist external-import gate (#4008) - #4013

Merged
ranxianglei merged 2 commits into
masterfrom
2026-10-20_fix-mysql2-static-dist-crash
Oct 20, 2026
Merged

ranxianglei merged 2 commits into
masterfrom
2026-10-20_fix-mysql2-static-dist-crash

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Closes #4008

What's broken

Thin installs (a copy of dist/ + package.json without a node_modules tree — dsh profile bare copies, bili's own e2e rig) crash at proxy startup: ERR_MODULE_NOT_FOUND: Cannot find package 'mysql2' imported from .../dist/index.js. The port never binds and model traffic silently goes direct/uncompressed. In CI this showed up as the two red assertions in real dsh headless profile lane (#2241 A / #2791 C "both turns stamped ONE conversation id" — actually 0, because no request ever reached bili).

Root cause

Fix (two commits)

  1. Lazy driver load (fix:) — the driver now loads only inside openBackend("mysql", dsn): a literal dynamic import of an external specifier survives bundling as a real runtime import (same discipline as the jimp lazy load, v0.1.198 release blockers found in final pre-release review | v0.1.198 发版前终审发现的阻塞项清单 #3516), and the resolved module is injected into MysqlBackend's constructor. Boots that select sqlite never resolve mysql2. A thin install that selects mysql fails at open time with a contextual error that the existing documented degrade paths already handle (shadow: warn + legacy-only; export: notice + sqlite fallback).
  2. Gate generalization (ci:) — check-dist-external-zod.mjs → scripts/check-dist-external.mjs: scans every dist artifact for ANY external bare import (all static forms incl. minified import{x}from"y" / export-from / require, plus dynamic import()) and fails unless on an explicit allowlist with a reason AND a sanctioned kind set: jimp + mysql2/promise are dynamic-only, react confined to agent/dsh-native-client.js. Both directions pinned in tests (Test Blind Spot Retrospective: Reinforcing Negative Assertion Discipline, Tightening Fake Upstream Strictness, Thin Inst | 测试盲区复盘:补反向断言纪律、假上游严格化、薄安装冒烟、形态矩阵与生命周期覆盖 #3617): the exact dist/index.js crashes on startup: mysql2 is statically inlined into the bundle, so deployment modes without node_modules | dist/index.js 启动即崩:mysql2 被静态打进 bundle,不带 node_modules 依赖的安装形态(dsh e2e、dist-only 嵌入)直接挂 #4008 minified shape fails, every zod form still fails (197-master 发版最后 #3511 pin preserved), a STATIC form of an allowlisted dynamic-only entry still fails, unknown new packages fail in all six forms, every sanctioned shape passes.

Behavior changes (old → new)

Case Old New
Thin install, any backend selection process dies at module load (ERR_MODULE_NOT_FOUND), port never binds boots cleanly; sqlite fully functional
Thin install, storage.backend=mysql selected same startup crash (before DSN validation even ran) openBackend("mysql") rejects with a contextual message naming both knobs; existing degrade paths apply
Normal install, mysql selected pool created at open time from the already-loaded module identical semantics (driver resolves from node_modules at open time)
Build-gate coverage literal "zod" only any external bare import outside the allowlist, incl. static forms of allowlisted lazy deps

No config-surface change. No kernel change. No wire change (storage layer only; both compression modes unaffected).

Verification

  • npm run typecheck clean; targeted suites green: tests/check-dist-external.test.ts (both directions), tests/storage-backend-knob.test.ts, tests/storage-backend-contract.test.ts, tests/cli-export.test.ts, tests/storage-unified.test.ts, tests/storage-backend-mysql-e2e.test.ts (DSN lanes skipped locally — no MySQL server here).
  • Rebuilt dist scanned: mysql2/promise now appears dynamic-only in dist/index.js + dist/agent/pi-native.js; zero static externals anywhere else (jimp dynamic, react confined to the browser bundle); new gate passes the build.
  • Thin-copy smoke (the issue's exact repro shape): copied dist/ + package.json into a scratch dir with no node_modules → node dist/index.js --version exits 0 and the proxy binds its port (before: immediate ERR_MODULE_NOT_FOUND).
  • real dsh headless profile lane needs dsh ≥ 0.2.0-rc.2 (local binary is older), so the two red assertions are CI's call on the pushed head.

awork added 2 commits October 21, 2026 00:14
… self-contained (#4008)

src/storage/mysql.ts held a TOP-LEVEL static value import of mysql2/promise. openBackend's dynamic import of ./mysql.js is an internal-module import that tsup's single-file bundle (splitting:false) hoists into the bundle body, and mysql2 (a package.json dependency) is auto-externalized by tsup, so the driver import landed as a static import at the top of dist/index.js and dist/agent/pi-native.js. ESM resolves static imports at module load regardless of use: every install without a node_modules tree (dsh profile bare copies, bili's own e2e rigs) crashed at startup with ERR_MODULE_NOT_FOUND before the proxy bound its port, and model traffic silently went direct, uncompressed (the 'real dsh headless profile lane' reds on #4003 ci:full).

The driver now loads only inside openBackend("mysql", dsn): a literal dynamic import of the external specifier survives bundling as a real runtime import (same discipline as the jimp lazy load, #3516), and the resolved module is injected into MysqlBackend's constructor. Boots that select sqlite never resolve mysql2; a thin install that selects mysql fails at open time with a contextual error that the existing documented degrade paths already handle (shadow: warn + legacy-only; export: notice + sqlite fallback), instead of dying at process start.
scripts/check-dist-external-zod.mjs only matched the literal bare specifier 'zod', so the mysql2 static import from #4008 sailed through every build. The gate now scans every dist artifact for ANY external bare import (all static forms incl. minified import{x}from"y" / export-from / require, plus dynamic import()) and fails unless the specifier is on an explicit allowlist whose entries carry a reason AND a sanctioned kind set: jimp + mysql2/promise are dynamic-only (their lazy-load contract is what keeps thin installs alive), react is confined to agent/dsh-native-client.js (dsh web-profile browser bundle). Node builtins and relative paths stay exempt; opaque variable specifiers remain invisible by design (the sanctioned host-zod path).

Both directions are pinned in tests/check-dist-external.test.ts (#3617 discipline): the exact #4008 minified shape fails, every zod form still fails (#3511 pin preserved), a STATIC form of an allowlisted dynamic-only entry still fails, unknown new packages fail in all six forms, and every sanctioned shape passes. Renamed to scripts/check-dist-external.mjs to match its actual scope; build wiring + stale references updated.
@ranxianglei ranxianglei changed the title fix(dist): load mysql2 lazily at backend-open time + generalize the dist external-import gate (#4008) [PR] fix(dist): load mysql2 lazily at backend-open time + generalize the dist external-import gate (#4008) Oct 20, 2026
@github-actions

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-20_fix-mysql2-static-dist-crash (d2df006)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.4793-pr.4013.c49346e

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr4013.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei
ranxianglei merged commit ab7e941 into master Oct 20, 2026
37 of 41 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.689-master.ab7e941 (rolling master tag).

npm install -g billion-context@0.0.689-master.ab7e941  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant