Repository navigation
[PR] feat(perf): #5460(P2) bili bench + fixture generator — record-only perf noise floor - #5477
ranxianglei wants to merge 20 commits into
Conversation
📦 Built Package ArtifactBranch: Latest build — install this exact versionnpm install -g billion-context@0.0.10263-pr.5477.81fb583This exact version is immutable and reproducible. The rolling Recent builds (commit → version)
Download artifact
tar xzf billion-context-pr5477.tgz
npm install -g packageInstall instructions are refreshed on each push. |
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) 查重:不重复 —— 检索 open/closed 的 issue 与 PR(bench / fixture-gen / perf noise floor 关键词):无其他在途 PR 实现 标签: 审核进行中——已拉取分支(对 master 全栈 12 文件 +1284/-7;P1 合入后的有效 delta = 5 文件 +799/-6),将按八问体例给出结论。 |
(ai: vllm-qwen/qwen3.8-27b) Review 结论(独立重审 effective delta + P1 接线)审核方式:不照抄 OP 描述——逐段重读了 commits 2–3 的完整 diff(
发现并已修(已推两个标注 commit 到本分支):
修后本地 结构化八问
判定:代码质量达标(测试断言有牙,e2e 真跑通),文档缺口已由我在分支上补齐。合并顺序请按 OP 说的:先合 P1 (#5461),再合本 PR(本分支以 P1 head 为祖先,栈一致)。等新 head CI 转绿我给最终确认。 |
ac795f4 to
e2b9043
Compare
(ai: vllm-qwen/qwen3.8-27b) Conflict-rebase complete — 1. What was broken? 2. Why did it conflict?
3. How was it resolved?
4. What does the diff look like now? 5. Was it verified? 6. Any behavior change introduced by this run? 7. What's next? |
[ework] CI fix verified green ✓ |
e2b9043 to
242ebc2
Compare
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) Conflict-rebase + red-CI fix complete — 1. What was broken? 2. Why did it conflict / why was CI red?
3. How was it resolved?
4. What does the diff look like now? 5. Was it verified?
6. Any behavior change introduced by this run? 7. What's next? |
…only, serving-immune) - src/perf-telemetry.ts: perf_events table (WAL, retention cap 200k rows, ring buffer), kill switch BILI_PERF_TELEMETRY, pure-JSON mode frozen/off, degrade-not-throw discipline (#5327 culture) - src/server.ts logRequestCost: structured mirror row through the single observation point every lane already flows through (#5408) - src/persist.ts: arm sqlite/mysql lanes at the two sidecar-open paths, explicit json-mode off - LIFECYCLE.md §2.4: BILI_PERF_TELEMETRY registered - tests/issue5460-perf-telemetry.test.ts: 8 cases incl. pinned DDL/WAL golden
The open-failure and insert-failure one-time warnings fell back to raw console.error, which bypasses bili's tee logger. For launcher-spawned children (#2864) stderr is physically redirected to a temp port-log, so the single 'telemetry degraded' signal never reached the main bili.log - exactly the observation surface this feature exists to feed. Route all degrade paths through loggerLog (the same channel the mem-sample path already used). No behavior/serving change; AGENTS no-console.error-in-server-modules.
The new env-only kill switch was registered in LIFECYCLE.md but missing from CONFIGURATION.md en+zh, violating the 'document or don't ship' env-var discipline (reference/env-vars.md hard rule 4). Every comparable env-only knob is listed there; add it to the Environment-only variables table in both languages.
…n, zero footprint The sqlite arming branch runs for EVERY enabled store, including storage.unified=legacy deployments where maybeOpen() returns null — so a pure-JSON install would still create perf.sqlite on first record, violating owner constraint ② (#5460: JSON mode frozen). Gate both DB branch arming sites on the RESOLVED mode: legacy → init("json") (permanently inert), shadow/full → arm as before. A configured DB-mode store whose sidecar transiently failed to open still arms (observability is most needed while degraded); only an explicit legacy choice freezes.
…en failure ensureDb() set state.dead on open failure but never checked it back, so a permanently broken dir retried new DatabaseSync on EVERY request and every 30s mem tick — contradicting the module's documented contract (one warn + permanent stop, serving untouched). Add the dead check at the top. The open-failure warn keeps the tee-logger routing landed upstream (9438e41) and additionally honors the caller-supplied logger (state.log) so tests can pin the one-shot behavior; production callers pass baseLog either way. Regression pins (both previously uncovered — the serving-immunity claim had no negative-direction test at all): * open failure (garbage file at the db path) → exactly ONE degrade warn, no throw through perfRecordRequest/perfRecordMemSample, no rows recorded; * storage.unified=legacy SessionStore → arms the frozen engine, creates NO perf.sqlite (wiring-level pin of the previous commit's constraint ② gate). Also drops the empty before() hook left in the test file.
…h fsync parity with index.db) WAL was set but synchronous stayed at the SQLite default FULL: every request row pays an fsync per commit on the request path. index.db sets journal_mode=WAL + synchronous=NORMAL together (src/storage/schema.ts); match that pair and pin the pragma in the schema golden test.
…ogRequestCost The PR's telemetry line and the pre-existing human-log line each ran Buffer.byteLength(outbound) — a second O(n) scan over potentially large response strings on every request. Hoist to one const.
The amortized sweep (DELETE ... ORDER BY id DESC LIMIT -1 OFFSET cap every 500th insert) is what keeps perf.sqlite bounded per owner constraint ① (crash/size isolation from index.db); an off-by-one in that SQL would let the file grow unbounded with zero other signal. Drive 200_001 real inserts through the public API and assert exactly 200_000 rows remain, newest kept.
… boundary The previous commit asserted exactly 200_000 rows after 200_001 inserts, which lands one insert past the last sweep boundary: the amortized sweep (every 500th insert) legitimately leaves up to cap+period-1 rows between sweeps, so 200_001 is correct behavior, not a cap violation. Drive to an exact multiple of the sweep period past the cap (200_500) so the final sweep deterministically trims the table to precisely the 200k window.
… driver seam — CLI stays loadable on Node <22.5
…ace-guard (#5460 P2) The six new CLI flags (--rounds/--out/--keep-workdir/--dist/--seed/--msgs/--shape) had no mention in README.md or CONFIGURATION.md, which is exactly what the config-surface-guard CI job blocks on (AGENTS.md Configuration Surface Discipline). Adds both subcommands + their flags to the CLI reference tables (en+zh) and a short performance-tooling section in both READMEs.
The help text for bili plugin-register / bili --version was shifted 2->3 spaces as an incidental whitespace edit; restored to master's bytes so the diff carries no whitespace churn on pre-existing lines (new bench/fixture-gen lines keep the subcommand-group indent used by the sessions/storage groups).
…he table row is generated (#5460 P1) check-coverage requires every process.env read in src to have a seed entry in website/config-reference/*.yaml, and docs-sync regenerates the CONFIGURATION tables strictly from those seeds. P1 hand-wrote its row into both files instead, so regeneration deleted it -> drift -> red docs-coverage + docs-sync on the stacked head. Seed added with P1's exact wording; regenerated row lands at the same table position (default column normalized to the seed's "on"). EN file ends byte-identical to before; zh picks up the normalization.
…ports (#5460 P2) Six types (BenchScenario/BenchStat/BenchRunReport/BenchArtifact/ BenchOptions/FixtureSpec) are exported but never imported outside their own file — the gate classifies that INTERNAL_ONLY. Tests consume only the value exports (computeStats, BENCH_SCENARIOS, generateFixture, render*Body); no public API surface changes.
…5460 P1) Windows cannot unlink a file with an open SQLite handle. P1 telemetry opens perf.sqlite in the store's sessions dir; a short-lived store closing without releasing it pins the caller's rmrf (EBUSY, win22 shards). Release the handle in close/closeQuietly/closeAsync via perfTelemetryReleaseIfBound (same contract index.db already follows). No-op for a foreign dir, idempotent.
242ebc2 to
ee49ab5
Compare
[bot] (ai: vllm-qwen/qwen3.8-27b) Re-rebased the PR onto the new master tip and resolved every conflict. 七问:
|
…ed BILI_STORAGE_UNIFIED/BILI_ENCRYPTION_KEY (#5460 P1) The test asserts the store arms sqlite telemetry on its own dir, but its resetModule() only cleared BILI_PERF_TELEMETRY — not BILI_STORAGE_UNIFIED or BILI_ENCRYPTION_KEY. When a prior test file in the same shard leaks one of those vars, the store arms in inert json mode (or is blocked by encryption) and perfBoundDir() returns null. Save+delete both vars before constructing the store, restore in finally — same pattern the legacy-footprint test already uses.
…n the file map (#5460) The architecture-docs guard (tools/check-architecture-docs.mjs) requires every tracked src/ file to be listed in reference/architecture.md. The three new perf modules landed without map entries, tripping the guard.
[bot] (ai: vllm-qwen/qwen3.8-27b) 七问 1. What was broken 2. Why conflict+red 3. How resolved
4. Diff now
5. Verified
6. Behavior change 7. What's next |
…gen hint, gated bench dist Small review nits on the P2 bench/fixture surface: - src/cli.ts: the two new help lines (bili bench / bili fixture-gen) were indented 3 spaces while every other command line uses 2 — realigned. - src/cli.ts: removed a doubled blank line before the -i/-s posture block and a stray blank line after parseArgs' return. - src/cli.ts: the fixture-gen missing-args error suggested 'bili bench <scenario>', which is not an alternative for missing fixture-gen args — replaced with a concrete 'bili fixture-gen --seed 42 --msgs 50' example. - tests/issue5460-perf-bench.test.ts: the gated e2e smoke called runBench without a dist, so resolveDist fell back to process.argv[1] (the test file itself under 'node --test') and the proxy could never boot. Pass the built dist explicitly (E2E_BILI_DIST ?? ../dist/index.js), matching the e2e lanes' convention. Verified: ACP_TEST_BENCH=1 now goes 10/10 without E2E_BILI_DIST. - CHANGELOG.md: collapsed three consecutive blank lines to one.
|
(ai: vllm-qwen/qwen3.8-27b)
Reviewed end-to-end: full diff (17 files, +1401/−6) — the P1 telemetry core ( Verdict: APPROVE. The tool is record-only and purely additive: new 已顺手修:help 两行缩进(3 空格→2 空格对齐惯例)、 八问(PR review 结论,中文)
判定:可合(APPROVE)。 不碰 |
(ai: vllm-qwen/qwen3.8-27b) Review complete — verdict APPROVE (posted on the GitHub PR as a comment; GitHub blocks a formal approve on the author's own PR). Fixed 5 small findings on-branch (commit |
(ai: vllm-qwen/qwen3.8-27b) Status check triggered by the review echo — no new review content; the APPROVE above stands. Current state of the merge chain, verified live just now:
Nothing left in my lane (P2 review): code verified, CI green, review posted. The chain now waits on: ① P1's conflict pass landing (its own thread/lane), ② owner approval + merge of P1, ③ owner merge of this PR. 判定:本 PR 侧无遗留工作——CI 全绿、审核 APPROVE 已发;合并链当前卡在 P1 #5461 的冲突(其线程由 CI 车道处理,11-15 19:12Z 接管后无完成记录),待 P1 合入后本 PR 即可由 owner 直接合并。 |
Stacked on #5461 (P1,
perf/l1-sqlite-telemetry) — please merge P1 first; after it lands, this PR's effective delta is commits 2–3.Commits
import "node:sqlite"crashes the entire CLI at module load on Node <22.5 while package.json still declares engines >=20, and it trips the dist self-containment gate. It now resolves through the same engine ladder as index.db (openSqliteDatabase: builtin node:sqlite → host better-sqlite3 → self-download, 彻底解决 dsh/pnpm 构建脚本门导致 build-blocked:SQLite 引擎自举(内置优先 → 运行时自下载 → 复用宿主) #2833/feat: SQLite engine self-bootstrap — builtin -> host -> cached -> self-download (#2833) #2845); when no engine exists at all it degrades per the existing contract — one warn, telemetry off, serving untouched. Zero new external specifiers.What P2 delivers (tracking issue #5460 roadmap, direction owner-approved 2026-11-01)
bili fixture-gen --seed K --msgs N --shape openai|anthropic [--out FILE]sessions.hardMaxBytes(Windows, 0.1.210) #5403 shape); big data is generated on demand, never committed.bili bench <scenario> [--rounds N] [--out FILE] [--keep-workdir] [--dist PATH]total_msp50/p95/max/min/mean, inbound/outbound byte totals, mem HWM (rss/heap), round error list;--rounds 3median support is already in place).smoke-openai-tiny,proxy-append-openai-s-small,proxy-append-openai-s-med,proxy-append-anthropic-s-med,proxy-compress-openai-s-med(declared 16K window forces fold density),proxy-append-openai-s-large(nightly tier).The OP named the generator
gen-perf-fixture; single-bin packaging makes abilisubcommand the zero-packaging-change form of it (no new tsup entry, ships in dist/index.js).Verification (targeted locally; full suite is CI's job)
npm run typecheck+npm run buildgreen including the dist self-containment gate;tests/issue5460-perf-bench.test.ts9/9 ungated + gated e2e smoke (ACP_TEST_BENCH=1, prebuilt dist) 1/1;tests/issue5460-perf-telemetry.test.ts20/20 (P1 suite transparent to commit 1);tests/cli-parseargs.test.ts7/7;bili bench proxy-append-openai-s-small: 50 rounds, 0 failed, req_rows=50, total_ms p50=5ms p95=9ms (max 2725ms = first-round cold start), inbound 5.3 MB total.Behavior changes & config surface
BILI_PERF_TELEMETRYis P1's surface.)