Skip to content

[PR] perf: bound the initial context profile sample - #6656

Merged
ranxianglei merged 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-14_bounded-context-profile
Nov 14, 2026
Merged

ranxianglei merged 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-14_bounded-context-profile

Conversation

@Afloat16

@Afloat16 Afloat16 commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Closes #6655

Problem

The initial context profile retains 100 message samples, but first allocates and sorts a second array as large as the complete history. This makes a bounded observability feature add avoidable temporary memory and synchronous latency to a long session's first request.

Change

Keep the largest 100 entries in a sorted bounded sample as messages are scanned. Reject candidates below the cutoff before allocating a sample entry, insert equal-size entries after earlier ones, and continue computing token and character totals over the entire request. The returned profile and first-observation behavior are preserved.

Validation

  • All 13 tests in tests/context-analysis.test.ts pass, including a mixed 513-message history checked against a full-sort oracle, thinking-token ranking, complete aggregate totals, and 250 equal-sized messages across the cap.
  • Strict TypeScript checking of src/context-analysis.ts and tests/context-analysis.test.ts passes with --target ES2022 --module NodeNext --moduleResolution NodeNext --strict --skipLibCheck --noEmit.
  • npm run build and git diff --check pass.
  • A fresh-process synthetic 1,000,000-message run on Node 24.22.1 measured heap growth of 117.85 → 4.01 MiB during sampling and elapsed time of 588.7 → 58.1 ms. Both runs returned the same aggregate counts and largest-100 sample. The issue contains the reproduction. These single-run measurements do not establish a general production speedup.

The complete repository npm run typecheck could not finish in this environment: the process was killed at the default/2048 MiB settings and exhausted the V8 heap at lower limits. The focused strict check succeeded; the complete typecheck and full-suite verdict remain for CI on the published head.

Scope and base

No kernel, compression policy, configuration knob, dependency or wire behavior changes.

  • Upstream base: e5e1647ba0f92dfc36a96220f3e5c0d454218e2f.
  • Head: 3be4ed07567a5bf901f1563bb86a0f14fe4f1552.
  • Published tree 29df36873efe7c707b1ceced5adea9ae24ae3f58 matches the tested local tree.

Hosted CI follow-up

On CI run 43565791430 for the published head, all Linux test shards passed. The Windows 22 and Ubuntu 24 static gates also passed, including the complete repository typecheck and build.

Two Windows test shards are red, so the overall CI result is not green:

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复 — #6656 是 #6655 的唯一 PR,无并列 PR 修同一问题。

Review of #6656 — perf: bound the initial context profile sample

I read the diff, traced the algorithm, and verified it independently (not from the PR description).

Direction (①) — correct. The issue is real and framed at the right layer: snapshotInit() built the full per-message array (one object per message, N of them) and sorted it (O(N log N)) only to keep the top 100. On a long session's first request that added O(N) temporary heap + O(N log N) synchronous work to a strictly-advisory profile (no decision path reads it — src/context-analysis.ts:19). The PR's direction — bound the sample during the scan instead of build-then-truncate — is the right root-cause fix (removes the producer, doesn't filter the trace downstream).

What I verified (③/④):

  • Traced the binary-search insertion: it keeps a non-increasing array of ≤100 entries, inserts a new equal-size entry after earlier ones (stable), and rejects a candidate only when the array is full AND tok <= the current 100th-largest. That is exactly "top-100, ties broken by arrival order" — what the original sort(desc, stable) + slice(0,100) produced.
  • Ran an independent full-sort oracle against the real code path on 309 inputs (300 random bounded up to 2000 msgs, strictly-increasing, strictly-decreasing, 1000 all-identical, tie-clusters crossing the cap, boundary n=99/100/101/102, thinking-token collisions): byte-identical perMessage in all 309.
  • Ran tests/context-analysis.test.ts on the head: 13/13 pass, including both new tests. The 513-message test is deepEqual against a full-sort oracle, so it has teeth (any set/order/tie/aggregate divergence fails it); the 250-equal test pins the "earlier wins ties" behavior.
  • Aggregates (messageCount, totalChars, estTokens) still cover the full request (the continue sits after the accumulations); perMessage is bounded at 100 (confirmed on a 1M-message run: perMessage.length === 100, messageCount === 1_000_000).

Regression (②) — none. The returned InitProfile is byte-identical to before (same 100 entries, same order, same tie-breaking, same aggregates); the only change is internal cost (less heap, less CPU). No observable behavior change, so no behavior-change flag. Contained to the pure function snapshotInit; no wire/config/kernel/persistence/message-id impact.

CI — 2 of 30 checks red: test (windows-latest, 22, 1) and (22, 3). I pulled both job logs: the failures are EBUSY: resource busy or locked, unlink …index.db in the bili-fpgate-* (shard 1) and bili-zstd-* (shard 3) tests — the documented Windows AV/indexer lock flake (#4347 family), not context-analysis. Master (#6643) is green on the same shards, and the PR's files are pure cross-platform in-memory logic (no file I/O / timing / process spawn), so these are flaky, not a regression. A re-run should clear them.

Quality (②a/⑤b, external submission — strict bar) — high. Small focused diff (2 files, 1 commit), correct algorithm, real oracle-based tests, reuses the existing INIT_PER_MSG_CAP constant (no magic number), no comments added, style matches the file, no new deps, and an honest description that explicitly caveats the single-run perf numbers. No restructuring needed.

Base — not stale: master advanced only touched src/update.ts + its test, not the PR's files; clean merge onto current master.


结构化八问

  1. 是什么问题:snapshotInit()(context-analysis 首请求可观测 profile)为了只保留 100 条样本,先把整段历史每条消息都分配成一个 entry 对象、再对整个数组排序——长会话首请求白白多出 O(N) 临时堆 + O(N log N) 同步开销,而这 profile 是纯 advisory(无决策路径读它)。
  2. 什么原因导致的:实现是「先全量构建样本再截断」,根因在扫描循环里,不在展示层。
  3. 影响范围与严重程度:仅长会话首请求的临时内存与同步延迟(advisory 功能);不影响任何决策/压缩/线上行为。严重度低(性能/内存优化,非可用性缺陷)。
  4. 怎么修复的:扫描时用排序的有界 top-100 数组边扫边维护——二分找插入位、等大的插到更早的后面(稳定)、满 100 且候选 ≤ 当前第 100 大时直接 continue(不分配对象)。从源头去掉 O(N) 分配 + O(N log N) 排序。
  5. 是否有回归问题:无。返回的 InitProfile 与改前逐字节一致(同 100 条、同序、同 tie 裁决、同聚合值)——独立全排序 oracle 对 309 组输入验证 perMessage 逐字节相同,13/13 测试通过。改动局限于纯函数 snapshotInit,不触 wire/config/kernel/持久化/消息 id。CI 两个红是 Windows EBUSY unlink index.db 已知 flake(fpgate/zstd 测试,非本 PR 文件),master 同 shard 绿,重跑即清。
  6. 是否破坏缓存:不涉及。纯内存可观测,无缓存/重试/游标/去重键。
  7. 是否改了设计方向:没有。仍是「内存内、有界、纯 advisory」。
  8. 是否引入了新的配置:没有。

判定:可合(approve)。 代码正确、质量高、测试有牙、diff 干净;唯一 caveat 是两个 Windows flake(与本 PR 无关,重跑即绿)。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Initial context analysis allocates and sorts the whole history to retain only 100 samples

2 participants