Skip to content

[PR] feat(perf): #5460(P2) bili bench + fixture generator — record-only perf noise floor - #5477

Open
ranxianglei wants to merge 20 commits into
masterfrom
perf/l2-bench-fixtures
Open

ranxianglei wants to merge 20 commits into
masterfrom
perf/l2-bench-fixtures

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Stacked on #5461 (P1, perf/l1-sqlite-telemetry) — please merge P1 first; after it lands, this PR's effective delta is commits 2–3.

Commits

  1. fix(perf): resolve the SQLite engine through the shared storage driver seam — already pushed directly onto the P1 branch (it belongs to [PR] feat(perf): realtime telemetry into perf.sqlite — DB modes only, serving-immune (#5460 P1) #5461). P1's top-level static import "node:sqlite" crashes the entire CLI at module load on Node <22.5 while package.json still declares engines >=20, and it trips the dist self-containment gate. It now resolves through the same engine ladder as index.db (openSqliteDatabase: builtin node:sqlite → host better-sqlite3 → self-download, 彻底解决 dsh/pnpm 构建脚本门导致 build-blocked:SQLite 引擎自举(内置优先 → 运行时自下载 → 复用宿主) #2833/feat: SQLite engine self-bootstrap — builtin -> host -> cached -> self-download (#2833) #2845); when no engine exists at all it degrades per the existing contract — one warn, telemetry off, serving untouched. Zero new external specifiers.
  2. feat(perf): Perf observability: realtime telemetry (SQLite-backed), web panel page, FULL/nightly CI bench gates #5460(P2) bili bench + fixture-gen — the P2 deliverable.
  3. test(perf) — determinism/report-math/CLI-parse coverage.

What P2 delivers (tracking issue #5460 roadmap, direction owner-approved 2026-11-01)

bili fixture-gen --seed K --msgs N --shape openai|anthropic [--out FILE]

bili bench <scenario> [--rounds N] [--out FILE] [--keep-workdir] [--dist PATH]

  • boots the REAL proxy (this very entry point) as a child process in throwaway XDG dirs on random ports (listen(0)) against an in-process deterministic fake upstream speaking both chat-completions and anthropic-message SSE (the e2e hermetic recipe);
  • replays the fixture as sequential full-resend rounds (the dsh plugin lane's traffic shape), then reads the child's perf.sqlite (Perf observability: realtime telemetry (SQLite-backed), web panel page, FULL/nightly CI bench gates #5460 P1) through the same driver seam as commit 1 and emits a JSON artifact: total_ms p50/p95/max/min/mean, inbound/outbound byte totals, mem HWM (rss/heap), round error list;
  • record-only, never judges: slow metrics still exit 0 — only harness failures (server never ready / every round failed / telemetry did not arm / mid-run crash) exit 1. Tolerance bands + baseline ratchet + FULL/nightly wiring are P3 (--rounds 3 median support is already in place).
  • six built-in scenarios: smoke-openai-tiny, proxy-append-openai-s-small, proxy-append-openai-s-med, proxy-append-anthropic-s-med, proxy-compress-openai-s-med (declared 16K window forces fold density), proxy-append-openai-s-large (nightly tier).

The OP named the generator gen-perf-fixture; single-bin packaging makes a bili subcommand the zero-packaging-change form of it (no new tsup entry, ships in dist/index.js).

Verification (targeted locally; full suite is CI's job)

  • npm run typecheck + npm run build green including the dist self-containment gate;
  • tests/issue5460-perf-bench.test.ts 9/9 ungated + gated e2e smoke (ACP_TEST_BENCH=1, prebuilt dist) 1/1;
  • tests/issue5460-perf-telemetry.test.ts 20/20 (P1 suite transparent to commit 1); tests/cli-parseargs.test.ts 7/7;
  • evidence run bili bench proxy-append-openai-s-small: 50 rounds, 0 failed, req_rows=50, total_ms p50=5ms p95=9ms (max 2725ms = first-round cold start), inbound 5.3 MB total.

Behavior changes & config surface

  • NO new env vars, NO new config fields — every knob is a CLI flag. (BILI_PERF_TELEMETRY is P1's surface.)
  • Behavior change from commit 1 (disclosed old→new): on Node <22.5, bili goes from "CLI dies at startup" (introduced by P1's static import, unshipped) back to "starts normally"; telemetry additionally runs on host better-sqlite3 where present instead of hard-requiring the Node builtin.

@github-actions

github-actions Bot commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: perf/l2-bench-fixtures (e3998e6)

Latest build — install this exact version

npm install -g billion-context@0.0.10263-pr.5477.81fb583

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
e3998e6 billion-context@0.0.10263-pr.5477.81fb583 build
fc115a9 billion-context@0.0.10258-pr.5477.7e0c1f5 build
c7a640e billion-context@0.0.10257-pr.5477.22add17 build
ee49ab5 billion-context@0.0.10253-pr.5477.89c66b8 build
242ebc2 billion-context@0.0.9628-pr.5477.e065fc5 build
e2b9043 billion-context@0.0.8942-pr.5477.3c2b182 build
ac795f4 billion-context@0.0.7767-pr.5477.0af34ee build
cb06f1b billion-context@0.0.7755-pr.5477.6fe4ce4 build
7f84ebe billion-context@0.0.7744-pr.5477.2e02c34 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr5477.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

查重:不重复 —— 检索 open/closed 的 issue 与 PR(bench / fixture-gen / perf noise floor 关键词):无其他在途 PR 实现 bili bench / bili fixture-gen。本 PR 是 #5460 路线图的 P2,叠在 P1 #5461 之上;已核实 P1 head f05e6e876 正是本分支的祖先(stack 一致,commit 1 确属 #5461)。

标签: pr + scope/ci + p3(性能测量工具面,record-only 无用户行为影响;非缺陷类不打 bug;不涉具体客户端车道故不打 client 标签)。

审核进行中——已拉取分支(对 master 全栈 12 文件 +1284/-7;P1 合入后的有效 delta = 5 文件 +799/-6),将按八问体例给出结论。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Review 结论(独立重审 effective delta + P1 接线)

审核方式:不照抄 OP 描述——逐段重读了 commits 2–3 的完整 diff(src/bench.ts 409 行、src/perf-fixture.ts、src/cli.ts 增量)、commit 1(driver seam 替换)、以及栈上 P1 的接线(src/persist.ts/src/server.ts/src/perf-telemetry.ts)。本地定向验证(全量回归归 CI):

  • npm run build 绿,含 dist 自包含门禁(dist-external: OK、dist-import-annotations: OK),build-info commit=PR head ✓
  • npm run typecheck 干净
  • tests/issue5460-perf-bench.test.ts + tests/issue5460-perf-telemetry.test.ts + tests/cli-parseargs.test.ts:27/27 过
  • gated e2e 实跑:ACP_TEST_BENCH=1 E2E_BILI_DIST=dist/index.js → 10/10,即真实代理子进程 + 假上游 + perf.sqlite 读链路端到端走通(不是只信单测)

发现并已修(已推两个标注 commit 到本分支):

  1. CI 红:config-surface-guard 失败(旧 head 上唯一真红,本地复现)。根因:guard(对现有配置文件的形态进行盘查,需要增加特殊ci专门屏蔽违规添加的配置 #2643)要求每个新 CLI 长 flag 必须出现在 README.md 或 CONFIGURATION.md,PR 新增的 --rounds/--out/--keep-workdir/--dist/--seed/--msgs/--shape 七个 flag 只在 --help 文案里,文档零登记。修法:CONFIGURATION.md/.zh-CN.md 的 CLI Reference 表补两条子命令 + 七个 flag 行,README.md/.zh-CN.md 各加一节「Performance tooling」(en+zh 同步)→ commit c6beb0dd7。
  2. 顺手小修:help 文本里既有的 bili plugin-register/bili --version 两行被顺带从 2 空格挪到 3 空格(无关行的纯空白漂移)→ 恢复 master 原字节,新行保留(subcommand 组缩进与 sessions/storage 组先例一致)→ commit cb06f1b8c。

修后本地 config-surface-guard 绿、cli-parseargs 7/7 绿。新 head cb06f1b8c 的 CI 在跑,转绿后我补一条确认。

结构化八问

  1. 是什么问题:这是功能 PR(Perf observability: realtime telemetry (SQLite-backed), web panel page, FULL/nightly CI bench gates #5460 P2 只记录型性能基准工具),不是缺陷修复;review 中唯一的"问题"是它的 CI 门禁 config-surface-guard 红着挡合并。
  2. 什么原因导致的:配置面纪律(guard)把 CLI flag 也纳入强制文档登记,PR 加了 flag 没登记——缺的是文档,不是代码。
  3. 影响范围与严重程度:仅挡本 PR 合并(GH 门禁红),运行时行为零影响,用户面零波及;属合并阻塞级、非功能级。
  4. 怎么修复的:按 guard 指定路径补齐文档(两 README + 两 CONFIGURATION,en/zh 同步),flag 语义逐条对照代码核实过(如 --rounds 默认 1、每轮独立记入 runs[];--dist ?? E2E_BILI_DIST ?? argv[1]),没有照抄 help 文案。
  5. 是否有回归问题:无。逐项核过的既有行为:① logRequestCost 日志行格式字节不变(outbound 长度只算一次,输出同形);② 遥测降级契约不变(open/insert 失败 → 一次性 warn + dead flag,服务路径不受影响);③ 存储模式默认仍是 full(Storage wrap-up: 6 fixes + switch default mode to full (single PR) | 存储收尾六修 + 缺省模式翻 full(一个 PR 统一处理) #3818),bench 显式 pin BILI_STORAGE_UNIFIED=full 与默认一致,注释属实;④ perf.sqlite 独立文件,不动 index.db schema/WAL;⑤ bench 子进程环境 hermetic 成立(BILI_/ACP_ 剥净 + 四个 XDG 变量全部重定位,paths.ts 优先 XDG 而非 homedir,Windows 同样收得住);⑥ maxPerfId 开库前先查存在性,不会提前造空库把「telemetry did not arm」守卫废掉;⑦ 无 kernel/ 改动、无版本号改动、无 pi-subagents 改动。
  6. 是否破坏缓存:不涉及——未触碰前缀缓存/折叠/reconcile 任何路径;compress 场景只是声明小窗口驱动折叠,走的仍是正常压缩管线。
  7. 是否改了设计方向:不涉及——纯增量观测工具,record-only 语义在代码里钉死(慢指标 exit 0,仅 harness 故障 exit 1),容差带/基线棘轮/FULL+nightly 接线按计划留 P3。
  8. 是否引入了新的配置:无新 env、无新配置文件字段(全是 CLI flag,现已按 guard 要求登记);BILI_PERF_TELEMETRY 属 P1 面,已在 CONFIGURATION/LIFECYCLE 登记。

判定:代码质量达标(测试断言有牙,e2e 真跑通),文档缺口已由我在分支上补齐。合并顺序请按 OP 说的:先合 P1 (#5461),再合本 PR(本分支以 P1 head 为祖先,栈一致)。等新 head CI 转绿我给最终确认。

@ranxianglei
ranxianglei force-pushed the perf/l2-bench-fixtures branch from ac795f4 to e2b9043 Compare November 10, 2026 13:12
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Conflict-rebase complete — perf/l2-bench-fixtures rebased onto master and force-pushed: ac795f4 → e2b9043 (16 commits replayed linearly). All conflicts resolved; no design decisions or authorization gates were involved. Seven-question report:

1. What was broken?
The PR head conflicted with its base (master) — merge state dirty. Master had drifted ~896 commits past the merge base since the branch cut.

2. Why did it conflict?
Four independent master-side moves collided with this PR's touch points:

3. How was it resolved?
Semantic-preserving relocation — every hunk kept the PR's intent, moved to its new home:

  • Telemetry call site (perfRecordRequest import + invocation) landed in src/server/request-cost.ts right after the wall-clock ms computation, where logRequestCost lives now; server.ts itself stays byte-identical to master.
  • The BILI_PERF_TELEMETRY lifecycle row went to docs/lifecycle.md §2.4 (after BILI_MYSQL_DSN); the env-table row + 7 CLI flag rows + 2 subcommand rows went to docs/configuration.md / .zh-CN.md in their respective tables; root stubs left untouched.
  • CHANGELOG entries (P1 Added section, driver-seam Fixed entry, P2 entries) re-inserted at the corresponding positions in the rewritten unreleased section.
  • cli.ts: union of master's posture block and the PR's bench/fixture fields in parseArgs' return.
    Two mechanical adaptations beyond raw conflict resolution (both folded back into their originating commits via fixup+autosquash):
  • tests/issue5460-perf-telemetry.test.ts:166 — late import path ../src/server.js → ../src/server/request-cost.js (the function moved; master's own test uses the same path).
  • Two bili bench / bili fixture-gen subcommand rows in the CLI reference tables that my first pass missed — caught by diff-stat comparison against the original PR and added.

4. What does the diff look like now?
15 files, +1330/−4 vs master — same content surface as before the rebase (new files src/bench.ts, src/perf-fixture.ts, src/perf-telemetry.ts + 2 test files unchanged in substance), only the relocated hunks differ in destination path. Zero leftover conflict markers (grep-verified across src/, docs/, website/, changelog, READMEs).

5. Was it verified?
On the pushed head: typecheck clean; npm run build green including dist self-containment gates (dist-import-annotations: OK, dist-external: OK); targeted suites tests/issue5460-perf-telemetry.test.ts + tests/issue5460-perf-bench.test.ts + tests/cli-parseargs.test.ts = 34 pass / 0 fail / 1 skip; config-surface-guard and lifecycle-ledger-guard both green (BILI_PERF_TELEMETRY registered, bench/fixture flags covered). Full-suite verdict belongs to remote CI.

6. Any behavior change introduced by this run?
None at runtime — this is a pure rebase. The telemetry wiring executes at the same logical point (inside logRequestCost, which now happens to live in request-cost.ts); all doc rows are the PR's own rows at their new canonical locations. No config keys added or renamed, no defaults touched. One timing note: master advanced mid-run (#5983, preflight view-divergence — files disjoint from this PR), so the final head sits on the fresh master tip a61c55cf rather than yesterday's; the second rebase replayed all 16 commits with zero conflicts. Push discipline held: remote head re-fetched and confirmed still ac795f4fde immediately before a --force-with-lease pinned to that exact SHA.

7. What's next?
Nothing further from this lane. CI will re-run on head e2b90435ed; the mirror check-cache had not yet picked up the new head at write time, so I'm deliberately not asserting any check state — watch the PR checks rather than this comment. If CI comes back red, I pick it up here (same bot owns the red-CI follow-through).

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[ework] CI fix verified green ✓

@ranxianglei
ranxianglei force-pushed the perf/l2-bench-fixtures branch from e2b9043 to 242ebc2 Compare November 13, 2026 17:13
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

Conflict-rebase + red-CI fix complete — perf/l2-bench-fixtures rebased onto current master and force-pushed: e2b9043 → 242ebc2 (16 PR commits replayed + 1 fix commit). Seven-question report:

1. What was broken?
Two things on the PR head e2b90435ed: (a) it was behind master — master had advanced 263 commits past the PR's merge base since the last rebase, and the new master's "pure-SQLite default" change (#6480) rewrote src/persist.ts in the same region the PR's P1 telemetry wires into; (b) all 5 windows-latest test shards were red while every Ubuntu shard was green.

2. Why did it conflict / why was CI red?

3. How was it resolved?

  • Rebased the PR's 16 commits onto the current master tip (e1f31ad9d5): clean replay, zero conflicts (the persist.ts hunks landed in non-overlapping regions and auto-merged with master's pure-SQLite-default change; telemetry wiring verified intact at src/persist.ts:42/703/728/732 and src/server/request-cost.ts:2/46).
  • Added the missing handle release: perfTelemetryReleaseIfBound(dir) in src/perf-telemetry.ts (no-op for a foreign dir, idempotent, keeps the armed mode) + one call in each of SessionStore.close() / closeQuietly() / closeAsync() in src/persist.ts, mirroring the index.db contract.
  • Force-pushed with --force-with-lease pinned to the verified remote head e2b90435ed (re-fetched immediately before; no other agent had pushed).

4. What does the diff look like now?
16 files vs master, +1392/−6. The PR's original 16 commits are byte-faithful (new src/bench.ts 409, src/perf-fixture.ts 108, src/perf-telemetry.ts, src/cli.ts +154, src/persist.ts, src/server/request-cost.ts, 2 test files, docs/README/CHANGELOG); the EBUSY fix is isolated in its own commit on top (3 files, +61/−3). Zero leftover conflict markers.

5. Was it verified?
Locally, targeted (full suite is CI's job):

  • tsc --noEmit clean.
  • tests/issue5460-perf-telemetry.test.ts 13/13 — incl. 2 new tests: releaseIfBound releases only the bound store's handle and a released handle never reopens; SessionStore.closeQuietly releases the perf.sqlite handle it armed.
  • tests/cli-parseargs.test.ts 14/14.
  • tests/issue5460-perf-bench.test.ts 9 pass / 1 skip (the gated e2e smoke needs ACP_TEST_BENCH=1 + a prebuilt dist).

6. Any behavior change introduced by this run?
One, and it is the point of the fix: after a SessionStore close, the process's perf.sqlite handle is now released (previously it stayed open for the process lifetime). A later store in the same process finds no binding and records nothing (no degrade, no retry) — the telemetry singleton is one-per-process by design, so this only affects short-lived/CLI stores that delete their dir. The serving path and the one-shot-degrade contract are untouched. No config keys, no defaults, no version bump, no kernel change.

7. What's next?
Live mirror cache now shows head 242ebc2aab in the CI queue: all 24 checks pending (0 failed so far), mergeable_state=blocked (checks in progress — will clear to clean once green). The 5 Windows shards should now pass (the EBUSY is fixed); Ubuntu was already green. I'm not asserting a final green — watch the PR checks. If anything comes back red I pick it up here (same bot owns the red-CI follow-through).

ework-agent and others added 17 commits November 16, 2026 03:06
…only, serving-immune)

- src/perf-telemetry.ts: perf_events table (WAL, retention cap 200k rows,
  ring buffer), kill switch BILI_PERF_TELEMETRY, pure-JSON mode frozen/off,
  degrade-not-throw discipline (#5327 culture)
- src/server.ts logRequestCost: structured mirror row through the single
  observation point every lane already flows through (#5408)
- src/persist.ts: arm sqlite/mysql lanes at the two sidecar-open paths,
  explicit json-mode off
- LIFECYCLE.md §2.4: BILI_PERF_TELEMETRY registered
- tests/issue5460-perf-telemetry.test.ts: 8 cases incl. pinned DDL/WAL golden
The open-failure and insert-failure one-time warnings fell back to raw console.error, which bypasses bili's tee logger. For launcher-spawned children (#2864) stderr is physically redirected to a temp port-log, so the single 'telemetry degraded' signal never reached the main bili.log - exactly the observation surface this feature exists to feed. Route all degrade paths through loggerLog (the same channel the mem-sample path already used). No behavior/serving change; AGENTS no-console.error-in-server-modules.
The new env-only kill switch was registered in LIFECYCLE.md but missing from CONFIGURATION.md en+zh, violating the 'document or don't ship' env-var discipline (reference/env-vars.md hard rule 4). Every comparable env-only knob is listed there; add it to the Environment-only variables table in both languages.
…n, zero footprint

The sqlite arming branch runs for EVERY enabled store, including
storage.unified=legacy deployments where maybeOpen() returns null —
so a pure-JSON install would still create perf.sqlite on first record,
violating owner constraint ② (#5460: JSON mode frozen). Gate both DB
branch arming sites on the RESOLVED mode: legacy → init("json")
(permanently inert), shadow/full → arm as before. A configured DB-mode
store whose sidecar transiently failed to open still arms (observability
is most needed while degraded); only an explicit legacy choice freezes.
…en failure

ensureDb() set state.dead on open failure but never checked it back, so a
permanently broken dir retried new DatabaseSync on EVERY request and every
30s mem tick — contradicting the module's documented contract (one warn +
permanent stop, serving untouched). Add the dead check at the top. The
open-failure warn keeps the tee-logger routing landed upstream (9438e41)
and additionally honors the caller-supplied logger (state.log) so tests can
pin the one-shot behavior; production callers pass baseLog either way.

Regression pins (both previously uncovered — the serving-immunity claim had
no negative-direction test at all):
* open failure (garbage file at the db path) → exactly ONE degrade warn,
  no throw through perfRecordRequest/perfRecordMemSample, no rows recorded;
* storage.unified=legacy SessionStore → arms the frozen engine, creates NO
  perf.sqlite (wiring-level pin of the previous commit's constraint ② gate).

Also drops the empty before() hook left in the test file.
…h fsync parity with index.db)

WAL was set but synchronous stayed at the SQLite default FULL: every
request row pays an fsync per commit on the request path. index.db sets
journal_mode=WAL + synchronous=NORMAL together (src/storage/schema.ts);
match that pair and pin the pragma in the schema golden test.
…ogRequestCost

The PR's telemetry line and the pre-existing human-log line each ran
Buffer.byteLength(outbound) — a second O(n) scan over potentially large
response strings on every request. Hoist to one const.
The amortized sweep (DELETE ... ORDER BY id DESC LIMIT -1 OFFSET cap every
500th insert) is what keeps perf.sqlite bounded per owner constraint ①
(crash/size isolation from index.db); an off-by-one in that SQL would let
the file grow unbounded with zero other signal. Drive 200_001 real inserts
through the public API and assert exactly 200_000 rows remain, newest kept.
… boundary

The previous commit asserted exactly 200_000 rows after 200_001 inserts,
which lands one insert past the last sweep boundary: the amortized sweep
(every 500th insert) legitimately leaves up to cap+period-1 rows between
sweeps, so 200_001 is correct behavior, not a cap violation. Drive to an
exact multiple of the sweep period past the cap (200_500) so the final
sweep deterministically trims the table to precisely the 200k window.
… driver seam — CLI stays loadable on Node <22.5
…ace-guard (#5460 P2)

The six new CLI flags (--rounds/--out/--keep-workdir/--dist/--seed/--msgs/--shape)
had no mention in README.md or CONFIGURATION.md, which is exactly what the
config-surface-guard CI job blocks on (AGENTS.md Configuration Surface Discipline).
Adds both subcommands + their flags to the CLI reference tables (en+zh) and a
short performance-tooling section in both READMEs.
The help text for bili plugin-register / bili --version was shifted 2->3 spaces
as an incidental whitespace edit; restored to master's bytes so the diff carries
no whitespace churn on pre-existing lines (new bench/fixture-gen lines keep the
subcommand-group indent used by the sessions/storage groups).
…he table row is generated (#5460 P1)

check-coverage requires every process.env read in src to have a seed entry
in website/config-reference/*.yaml, and docs-sync regenerates the
CONFIGURATION tables strictly from those seeds. P1 hand-wrote its row
into both files instead, so regeneration deleted it -> drift -> red
docs-coverage + docs-sync on the stacked head.

Seed added with P1's exact wording; regenerated row lands at the same
table position (default column normalized to the seed's "on"). EN file
ends byte-identical to before; zh picks up the normalization.
…ports (#5460 P2)

Six types (BenchScenario/BenchStat/BenchRunReport/BenchArtifact/
BenchOptions/FixtureSpec) are exported but never imported outside their
own file — the gate classifies that INTERNAL_ONLY. Tests consume only
the value exports (computeStats, BENCH_SCENARIOS, generateFixture,
render*Body); no public API surface changes.
…5460 P1)

Windows cannot unlink a file with an open SQLite handle. P1 telemetry
opens perf.sqlite in the store's sessions dir; a short-lived store closing
without releasing it pins the caller's rmrf (EBUSY, win22 shards). Release
the handle in close/closeQuietly/closeAsync via perfTelemetryReleaseIfBound
(same contract index.db already follows). No-op for a foreign dir, idempotent.
@ranxianglei
ranxianglei force-pushed the perf/l2-bench-fixtures branch from 242ebc2 to ee49ab5 Compare November 15, 2026 19:27
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] (ai: vllm-qwen/qwen3.8-27b)

Re-rebased the PR onto the new master tip and resolved every conflict. 七问:

  1. What was broken — the PR was in a merge-conflict ("dirty") state again: master moved 291 commits past the branch's base (e1f31ad → f648217), so it no longer merged cleanly.

  2. Why the conflict — master and the PR both edited the same spots since the base: (a) the "Environment-only variables" config table (docs/configuration.{md,zh-CN.md,ja.md} + website/config-reference/env-only.yaml) — master EXPANDED the BILI_CONFIG_FILE row (named-config fail-fast wording) while the PR ADDED a new BILI_PERF_TELEMETRY row right beside it; (b) src/cli.ts — master added a -c/--config fail-fast block in main() while the PR extended the parseArgs destructuring with the new bench/fixture fields.

  3. How it was resolved — rebased the PR's 17 commits (16 feature + 1 Windows-EBUSY fix) onto f648217, resolving each conflict as the union of both intents: keep the PR's BILI_PERF_TELEMETRY row AND master's expanded BILI_CONFIG_FILE row (per-language en/zh/ja); keep the PR's extended parseArgs destructuring AND master's -c/--config fail-fast block.

  4. Diff now — the branch is 17 commits on top of master f648217 (previously based on e1f31ad). Net change vs the old head (242ebc2) is just the conflict resolutions (config-table rows + the cli.ts main() block); the rest is master's 291 new commits coming in through the rebase.

  5. Verified — npx tsc --noEmit clean (after building the in-repo kernel + pi-subagents); targeted tests issue5460-perf-telemetry + cli-parseargs + issue5460-perf-bench = 37 tests, 36 pass / 0 fail / 1 skip (the gated e2e bench test, needs ACP_TEST_BENCH=1). The two Windows-EBUSY regression tests (releaseIfBound + closeQuietly) both pass. Force-pushed with --force-with-lease pinned to the old head: 242ebc2 → ee49ab5.

  6. Behavior change — none. The rebase only reconciles the conflict; no runtime behavior, default, threshold, timing, or output changes.

  7. What's next — CI is now running on the new head (ee49ab5): the merge-conflict gate is green (0 failed, remaining checks queued/pending). I'll watch it; if any shard goes red I'll fix it in this same lane (one bot owns both the conflict and the red CI).

ci-check added 2 commits November 16, 2026 04:23
…ed BILI_STORAGE_UNIFIED/BILI_ENCRYPTION_KEY (#5460 P1)

The test asserts the store arms sqlite telemetry on its own dir, but
its resetModule() only cleared BILI_PERF_TELEMETRY — not
BILI_STORAGE_UNIFIED or BILI_ENCRYPTION_KEY. When a prior test file in
the same shard leaks one of those vars, the store arms in inert json
mode (or is blocked by encryption) and perfBoundDir() returns null.
Save+delete both vars before constructing the store, restore in
finally — same pattern the legacy-footprint test already uses.
…n the file map (#5460)

The architecture-docs guard (tools/check-architecture-docs.mjs) requires
every tracked src/ file to be listed in reference/architecture.md. The
three new perf modules landed without map entries, tripping the guard.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] (ai: vllm-qwen/qwen3.8-27b)

七问

1. What was broken
Three test shards (ubuntu 22/24, windows 22) were red on the same test: #5460 P1: SessionStore.closeQuietly releases the perf.sqlite handle it armed (Windows EBUSY) at tests/issue5460-perf-telemetry.test.ts:264. The assertion "store construction arms telemetry on its own dir" failed: perfBoundDir() returned null instead of the sandbox dir. A fourth failure appeared after the test fix: architecture-docs guard — reference/architecture.md was missing entries for the three new src/ files (bench.ts, perf-fixture.ts, perf-telemetry.ts).

2. Why conflict+red
The test's resetModule() only cleared BILI_PERF_TELEMETRY — not BILI_STORAGE_UNIFIED or BILI_ENCRYPTION_KEY. When a prior test file in the same shard leaks one of those vars (e.g. BILI_STORAGE_UNIFIED=legacy), the store arms in inert json mode (or is blocked by encryption) and perfBoundDir() returns null. The test passed locally (clean env) but failed on all 3 CI shards (leaked env). The architecture-docs failure was a pre-existing gap: the PR added new src/ files without updating the file map.

3. How resolved

  • Made the test deterministic: save+delete BILI_STORAGE_UNIFIED and BILI_ENCRYPTION_KEY before constructing the SessionStore, restore in finally — same pattern the previous test in the file already uses.
  • Added the three missing entries to reference/architecture.md.

4. Diff now
Two new commits on the PR branch (no force-push):

  • c7a640e76d — test(perf): make closeQuietly-release test deterministic against leaked BILI_STORAGE_UNIFIED/BILI_ENCRYPTION_KEY (#5460 P1) (1 file, +9/-1)
  • fc115a967a — docs(arch): register bench.ts / perf-fixture.ts / perf-telemetry.ts in the file map (#5460) (1 file, +3)

5. Verified

  • Local: npx tsc --noEmit clean; tests/issue5460-perf-telemetry.test.ts 13/13 pass (also passes with BILI_STORAGE_UNIFIED=legacy and BILI_ENCRYPTION_KEY set — the leak scenarios).
  • CI: all 27 jobs green on head fc115a967a (run 43759651723), including the 3 previously red shards and architecture-docs.

6. Behavior change
None. Test-only + docs-only changes. No runtime behavior changed.

7. What's next
CI is green. PR is mergeable. Ready for review/merge.

…gen hint, gated bench dist

Small review nits on the P2 bench/fixture surface:
- src/cli.ts: the two new help lines (bili bench / bili fixture-gen) were
  indented 3 spaces while every other command line uses 2 — realigned.
- src/cli.ts: removed a doubled blank line before the -i/-s posture block
  and a stray blank line after parseArgs' return.
- src/cli.ts: the fixture-gen missing-args error suggested 'bili bench
  <scenario>', which is not an alternative for missing fixture-gen args —
  replaced with a concrete 'bili fixture-gen --seed 42 --msgs 50' example.
- tests/issue5460-perf-bench.test.ts: the gated e2e smoke called runBench
  without a dist, so resolveDist fell back to process.argv[1] (the test
  file itself under 'node --test') and the proxy could never boot. Pass the
  built dist explicitly (E2E_BILI_DIST ?? ../dist/index.js), matching the
  e2e lanes' convention. Verified: ACP_TEST_BENCH=1 now goes 10/10 without
  E2E_BILI_DIST.
- CHANGELOG.md: collapsed three consecutive blank lines to one.
@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: vllm-qwen/qwen3.8-27b)

Note: GitHub blocks a formal APPROVE on the author's own PR, so this review conclusion is posted as a comment. Verdict below is APPROVE.

Reviewed end-to-end: full diff (17 files, +1401/−6) — the P1 telemetry core (perf.sqlite + src/perf-telemetry.ts + persist/request-cost wiring) and the P2 bench/fixture surface (src/bench.ts, src/perf-fixture.ts, CLI subcommands), plus docs. No kernel/ or pi-subagents/ changes.

Verdict: APPROVE. The tool is record-only and purely additive: new bili bench / bili fixture-gen subcommands + a perf.sqlite beside the session dir, with a full kill switch (BILI_PERF_TELEMETRY=0) and zero footprint in JSON storage mode. The write path is fully isolated from serving (a write failure degrades telemetry with a one-shot warn and never touches serving). I verified the hermetic harness end-to-end (in-process fake upstream + real proxy child in a throwaway XDG dir, seeded full-resend replay, perf.sqlite read), the config routing that makes the 16K compress-window scenario work (longest-prefix route match on the origin form), and the SQLite driver seam (openSqliteDatabase) — all correct.

已顺手修:help 两行缩进(3 空格→2 空格对齐惯例)、parseArgs 一处双空行 + 一处多余空行、fixture-gen 缺参提示(原建议 bili bench <scenario> 不合逻辑,改为具体示例)、CHANGELOG 三连空行,以及唯一实质发现——gated e2e smoke 调 runBench 没传 dist,resolveDist 回退到 process.argv[1](node --test 下即测试文件本身)导致 proxy 永远起不来;现按 e2e 惯例显式传构建产物(E2E_BILI_DIST ?? ../dist/index.js)。已验证 ACP_TEST_BENCH=1 无需 E2E_BILI_DIST 即 10/10 通过。+ commit e3998e6

八问(PR review 结论,中文)

  1. 是什么问题:非修 bug,是 Perf observability: realtime telemetry (SQLite-backed), web panel page, FULL/nightly CI bench gates #5460 的 P2 功能——给 bili 加 record-only 性能基线工具(bili bench + fixture 生成器 + 实时 perf 遥测),为 [Bug][dsh] 20k-message / 81 MiB-inbound session: host aborted 6× in 2 h — the #3050 resident-AST OOM is not bounded by sessions.hardMaxBytes (Windows, 0.1.210) #5403 OOM / Long sessions starve the event loop: a 4,600-message session blocks the proxy and the web UI for 7–50 s per turn #4528 主线程停顿 / Long Claude Code session re-caches ~140-175K tokens per turn (cache read stuck at the fixed prefix) #5385 前缀钉死这类性能事故建噪声地板。
  2. 什么原因导致的:不适用(新功能,非缺陷)。动机:近期性能事故全靠用户先发现、CI 零预警,缺一个可复现的性能基准与遥测。
  3. 影响范围与严重程度:纯增量。新增 CLI 子命令 + 会话目录旁一个 perf.sqlite;不改任何既有行为;JSON 存储模式零足迹;BILI_PERF_TELEMETRY=0 可整体关闭。无外部影响。
  4. 怎么修复的(实现思路):hermetic 假上游(进程内 HTTP,识别 kernel 摘要 prompt)+ 一次性 XDG 目录里拉起真 proxy 子进程,把种子化确定性 fixture 按 full-resend 轮次重放,读 perf.sqlite 出 JSON artifact。record-only:慢指标仍 exit 0,仅 harness 失败(起不来 / 全轮失败 / 遥测没挂上 / 中途崩)才 exit 1。
  5. 是否有回归问题:无。既有行为逐条过——写路径与服务完全隔离(写失败只停遥测 + 一次性 warn,绝不影响服务);JSON 模式冻结零足迹;store close 时释放句柄(Windows EBUSY)。我顺手修的几处全是表面(help 缩进 / 空行 / 错误文案)+ 一个测试侧 dist 路径修正,均不改运行时行为。
  6. 是否破坏缓存:否。不动 cache / retry / 增量游标 / 去重键。perf.sqlite 是独立小库,WAL 模式,20 万行上限。
  7. 是否改了设计方向:否。沿用既有存储驱动缝(openSqliteDatabase)与 e2e hermetic 惯例,无架构转向。
  8. 是否引入了新的配置:是——新增 env BILI_PERF_TELEMETRY(默认开,=0 全关),已在 configuration.md en/zh/ja + env-only.yaml 记录;CLI 旗标 --out/--rounds/--keep-workdir/--dist/--seed/--msgs/--shape 为子命令作用域,非全局配置。

判定:可合(APPROVE)。 不碰 kernel/、无 kernel-change 声明,走正常 auto-merge 门即可。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Review complete — verdict APPROVE (posted on the GitHub PR as a comment; GitHub blocks a formal approve on the author's own PR). Fixed 5 small findings on-branch (commit e3998e673d), including the one real one: the gated e2e bench test called runBench without a dist so resolveDist fell back to process.argv[1] (the test file itself under node --test) and the proxy could never boot — it now passes the built dist explicitly (E2E_BILI_DIST ?? ../dist/index.js), verified ACP_TEST_BENCH=1 → 10/10 without E2E_BILI_DIST. The rest were cosmetic (help indentation, stray blank lines, a nonsensical fixture-gen hint, CHANGELOG blanks). No kernel/ or pi-subagents/ changes. Ready to merge (P1 #5461 first per OP).

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Status check triggered by the review echo — no new review content; the APPROVE above stands. Current state of the merge chain, verified live just now:

Nothing left in my lane (P2 review): code verified, CI green, review posted. The chain now waits on: ① P1's conflict pass landing (its own thread/lane), ② owner approval + merge of P1, ③ owner merge of this PR.

判定:本 PR 侧无遗留工作——CI 全绿、审核 APPROVE 已发;合并链当前卡在 P1 #5461 的冲突(其线程由 CI 车道处理,11-15 19:12Z 接管后无完成记录),待 P1 合入后本 PR 即可由 owner 直接合并。

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant