Skip to content

[PR] perf: align cache usage in a single ordered pass - #6722

Closed
Afloat16 wants to merge 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-14_linear-cache-usage-alignment
Closed

Afloat16 wants to merge 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-14_linear-cache-usage-alignment

Conversation

@Afloat16

@Afloat16 Afloat16 commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Closes #6721

Problem

The cache-diff report aligns each usage log sample by scanning backward from the last request. A long retained request/log history therefore performs requests × samples timestamp comparisons, even though both lists have already been sorted. This delays the offline diagnostic that identifies cache misses and their source.

Change

Advance one request cursor through the sorted timestamps while visiting usage samples. Each sample still belongs to the last request at or before its timestamp plus the existing 50 ms allowance. Equal-time requests, stable equal-time log ordering, the first sample used by a request, and missing samples retain their existing behavior. The timestamp-alignment step now takes O(requests + samples) work.

This change is confined to the offline cache report. It adds no configuration, storage format, dependency, kernel change, or request-pipeline behavior.

Current revision and validation

Rebased onto master a8bbb8d23108ba9148d09512927041364625d910 in e23468ac8c0162b646105f9fea135f20aafaa719, preserving the original regression and benchmark files and the upstream #6691 root-array correction.

On this exact revision, npm run typecheck and npm run build pass. All 50 tests across cache-usage alignment, cache-diff, root-array lookup, and Unicode suites pass with 0 failures or skips under TZ=UTC.

The main CI run was cancelled before the required test lanes completed. The local results above do not establish a green full-suite CI result.

Original verification

  • Eight new tests call public runDiff() with real dump/log files. They cover out-of-order log input, multiple samples per request, the exact 50 ms boundary and its rejected neighbor, duplicate dump timestamps, equal-time sample stability, session filtering, early/late samples, empty logs, and gaps.
  • These are compatibility tests: all eight already pass on the unchanged base. The improvement is measured separately.
  • 60/60 related tests pass with TZ=UTC, across cache-diff, Unicode attribution, new usage alignment, and log retention.
  • In this runner's default America/Los_Angeles timezone, the initial wider run was 58 passed / 2 failed. Both failures are unchanged log-retention fixtures that write UTC clock values without a timezone and expect a 50-hour span. Restoring the base source and rerunning those exact two tests reproduces both failures. The cache-diff tests pass in both environments.
  • Full npm run typecheck, separate strict TypeScript checking of the benchmark script, npm run build, and git diff --check pass.
  • Independent read-only review verified both sorted-input paths and the duplicate-time/first-sample semantics without a blocking finding.

Measurement

scripts/bench-cache-usage-alignment.ts creates 10,000 actual request dump files and 100,000 usage log samples, then calls the full public report path. Each process performs one complete warmup; fixture construction, assertions, and hashing stay outside the measured interval.

Three sequential alternating base/fix pairs on Node 24.22.1:

Public report Base Change
Median wall time 1,541.77 ms 434.95 ms
Median process CPU 1,756.29 ms 628.58 ms
Wall-time range 1,450.05–1,578.90 ms 397.16–492.88 ms

All six complete reports have the same SHA-256 (a0c63e939f3ec3333e8d56e207f9f41976b35b0d0078b57b0d423bbaa6c25aa0) after replacing only the temporary directory/log paths. The benchmark checks every pair's usage fields before hashing. These figures describe this synthetic filesystem workload; other report stages still sort and read files.

Reproduce one warmup plus one measurement at either revision:

node --import ./tests/preload-test-env.mjs --import ./tests/preload-storage-baseline.mjs --import tsx scripts/bench-cache-usage-alignment.ts 10000 10

History check

Reviewed all 127 Afloat16 issues and the separate 111-entry actual PR patch index, including current-batch cache-report changes. Actual #1269 introduces the existing backward scan; #5487 changes streaming log retention, #6496/#6517 fix Unicode byte offsets, and #6691 locates root message arrays. None changes usage alignment. Focused all-state issue/PR searches found no previous matching optimization.

Original benchmark base: master 3858b6956abab52da781c2436c0245586ce18358, package 0.1.214. Full-suite status comes from hosted CI on the pushed head.

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重: 不重复 — searched open+closed issues and PRs for usage-alignment / cache-diff alignment work: no prior matching optimization. The only related history is #1269, which introduced the backward scan this PR replaces (origin of the code, not a parallel fix).

标签建议: bug · scope/other · p2 — performance degradation in an offline diagnostic (not client-specific, not a live-path defect). Per the no-source-mutation rule I am not relabeling this thread myself; applying is left to the owner/maintenance.

Review

① Direction (original issue #6721). Correct, and it fixes the root cause, not a symptom. The producer of the quadratic work is alignUsage() re-scanning the whole request list per sample even though its single call site (src/acp-cache-diff.ts:623) receives both inputs already sorted ascending (g.out sorted at src/acp-cache-diff.ts:611; usage lists sorted in parseUsageLog at src/acp-cache-diff.ts:431 and in the default multi-file merge at src/acp-cache-diff.ts:588). With samples ascending, the "last eligible request" index is monotonically non-decreasing, so one forward cursor is exactly the right fix.

② Regression review. The diff touches one function in one offline-diagnostic module; nothing else in the 3-file diff. I verified behavior preservation, not just "tests pass":

  • Semantic equivalence: old code assigns each sample to the largest i with outTs[i] <= s.ts + 50ms; the cursor ends at exactly that index (monotonicity argument above). Edge cases checked: samples before the first request (unassigned in both), duplicate dump timestamps (both pick the last equal-time request), equal-time samples (stable), the 50 ms boundary (admitted at exactly −50 ms, rejected at −51 ms — both sides tested), missing samples (usageAvailable false).
  • The "compatibility tests" claim is real: I ran the 8 new tests against the unchanged base — 8/8 pass — and against the PR head 8/8 pass.
  • 37/37 tests pass across tests/acp-cache-usage-alignment.test.ts + tests/acp-cache-diff.test.ts + tests/acp-cache-diff-unicode.test.ts (TZ=UTC); npm run typecheck clean.
  • Output identity: I reproduced the bench at 10,000 requests × 100,000 samples on both revisions — the full-report SHA-256 is a0c63e93… in all four runs, matching the hash declared in the PR body.
  • High-risk patterns walked: no semantic drift, no fail-fast→swallow (the log-read throw in parseUsageLog is untouched), no threshold/timing/output-format changes (50 ms allowance unchanged), no config/storage/wire/kernel/pipeline surface touched.

③ Code. Two lines of logic + one comment. The added comment ("Both inputs are sorted by time, so each request need only be visited once") documents the precondition the new code leans on — good, since the old code silently leaned on outTs ordering too. No magic numbers (reuses USAGE_SKEW_MS), no new deps, style matches the file.

④ Test quality. These are real tests: public runDiff() path, real dump/log files, exact-field deepEqual assertions on the usage output. The −50/−51 pair is a proper two-sided gate (admits what it should, rejects what it shouldn't). For a perf change, pinning behavior with compatibility tests + measuring speed separately in a bench is the right design. The bench is rigorous: one full warmup per process, fixture/hashing outside the measured interval, all pair usage fields asserted before hashing, and the normalized-report hash as the equivalence check. I reproduced the improvement locally at 10k×10: base 1,115–1,316 ms vs change 389–407 ms wall (~3×), consistent with your 1,542→435 ms; the isolated loop alone goes 616 ms → 3.4 ms.

⑤/⑤b Diff & code cleanliness. 3 files, +151/−3, every line on-topic; no whitespace churn, no lockfile, no version bump. Naming clear, no dead code, no new style. Pass.

Findings (non-blocking)

  1. PR is 1 commit behind current master (a8bbb8d2, batch6-wave6 merge). Master's change to src/acp-cache-diff.ts (rootArrayStart, from [PR] fix: locate root message arrays in cache-diff attribution #6691) sits at lines 227–260, disjoint from the alignUsage hunk at 444–455 — no semantic interaction, but please rebase before merge.
  2. The two log-retention failures in your America/Los_Angeles run are pre-existing and disjoint. I reproduced both under TZ=America/Los_Angeles on the PR head (the "span reaching the target" and "unreadable generation" cases) and confirmed the retention tests share no code path with this diff; under TZ=UTC all 27 retention tests pass. Follow-up note (parked here per the no-new-issue rule, not filed): those fixtures write UTC clock values without a timezone and expect a 50-hour span, so npm test on a non-UTC local timezone fails on base — candidate for its own issue with a 来源 marker if the owner wants it tracked.

CI status (verified live, single fetch)

Static gates green on the pushed head: static-gates (ubuntu 24 + windows 22), architecture-docs, lock-sync, test-import-graph, version-guard, unused-exports, mysql-contract all success; e2e lanes correctly skipped (gated file lists not touched). The test shards (ubuntu/windows × node 22/24) were still queued/in-progress at check time — full-suite verdict pending on CI.

结构化八问

  1. 是什么问题:cache-diff 离线报告的 usage 日志对齐步骤是 O(requests × samples)——每个样本都从请求列表尾部整段回扫,长保留历史(万级请求)下诊断报告变慢(实测 ~1.5s)。
  2. 什么原因导致的:原实现对每个样本独立做完整反向扫描,但两个输入在唯一调用点都已按时间升序排序,「最后一个合格请求」的下标对样本序列单调不减——重复扫描是纯粹的浪费工作。
  3. 影响范围与严重程度:仅影响离线 cache-diff 诊断报告,不影响请求管线与真实缓存行为;只有保留历史很长(万级请求)的用户会感知到慢;属性能退化(p2),无功能错误。
  4. 怎么修复的:单一前进游标——遍历样本时只向前推进请求游标,样本仍归属「时间戳 ≤ 样本时间 + 50ms 的最后一个请求」;语义不变,复杂度降到 O(requests + samples)。
  5. 是否有回归问题:无。逐样本归属结果与旧实现数学等价(排序前提旧实现本就依赖,新注释使其显式);8 个新兼容性测试在旧代码与新代码上均 8/8 通过(本地在 base 上实跑确认);37 个 cache-diff 相关测试 TZ=UTC 全绿;新旧代码完整报告输出 SHA-256 一致(与 PR 声明的哈希相同)。高危模式逐项过:无语义漂移、无 fail-fast 变吞错、50ms 容忍度与边界行为(-50 收 / -51 拒)不变、无输出格式变化。
  6. 是否破坏缓存:不涉及——不触碰 prompt cache、重试、增量游标、去重键,只改离线报告的对齐计算。
  7. 是否改了设计方向:不涉及——无架构/边界/数据流变化,纯函数内算法优化。
  8. 是否引入了新的配置:不涉及——无新 env/config/默认值。

判定:可合——rebase 到当前 master 后、CI test 分片转绿即可合并。

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

CI update (verified live): 47/48 checks green on head c8b86a3f. The single red is test (windows-latest, 22, 4), and it is not caused by this PR:

  • Failing test: #5955 processBirthStamp: own pid resolves to a stable non-empty stamp (tests/issue5955-wedged-same-lane-takeover.test.ts:327) — a pre-existing master test in the wedged-lane takeover suite. This PR's diff touches only alignUsage in src/acp-cache-diff.ts plus the two new files; it shares no code path with the launcher.
  • Failure mode: the test took 12.8 s and the assertion got undefined. On win32, processBirthStamp (src/launcher.ts:5134) shells out to powershell -NoProfile … Get-Process with a 10 s timeout; a cold powershell start under the runner-pool saturation reported on the CI coordination thread a few hours ago blows that budget → undefined → the "birth stamp available on this platform" assertion fails. Environmental flake, not a code regression.
  • Action taken: I re-ran the failed job (run 43596236684); will report back once it lands.

Verdict unchanged: mergeable after rebase to current master + this shard re-run green.

@Afloat16

Copy link
Copy Markdown
Contributor Author

Rebased onto master a8bbb8d2 in e23468ac. On this exact head, npm run typecheck and npm run build pass, and all 50 cache-diff/alignment tests pass with 0 failures or skips under TZ=UTC. The original regression and benchmark files and the upstream #6691 root-array correction are preserved.

The main CI run was cancelled before the required lanes completed:
https://github.com/ranxianglei/billion-context/actions/runs/43601044339

Could you rerun the required CI on e23468ac? The PR description now records the rebased head and its local validation separately from the original measurements.

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Before re-running CI I checked the current state of master, and there's something you should know first: this PR's change is already on master, so a rerun on e23468ac would verify nothing that isn't already shipped.

  • The original head c8b86a3f (the one reviewed in detail earlier) was merged via 2221db1c9d — "Merge Afloat16/2026-11-14_linear-cache-usage-alignment (PR [PR] perf: align cache usage in a single ordered pass #6722) into batch7-wave7" — which is an ancestor of current master f21b6018.
  • Verified byte-for-byte against master: the alignUsage single-cursor implementation at e23468ac is identical to master's, and scripts/bench-cache-usage-alignment.ts + tests/acp-cache-usage-alignment.test.ts are byte-identical to master's copies.
  • e23468ac adds nothing beyond master — in fact it is behind it: since your rebase base (a8bbb8d2), master picked up newer changes to the same file (insertion-boundary work [PR] fix: keep whole-message inserts out of semantic appends #6738 and usage-log attribution tags [PR] fix: read logger attribution tags in cache usage #6742, via batch8-wave8).
  • Live CI state (just verified): the original head c8b86a3f is now 48/48 green — the one Windows flake I reran last night (#5955 processBirthStamp, an environmental powershell timeout, unrelated to this diff) passed on rerun.

So I'm not re-running CI on e23468ac: there is nothing left to merge, and the runner pool has been saturated all day. Suggested next step: since the work already shipped via batch7-wave7, you can close this PR yourself (I'm not permitted to close it). If you'd rather leave it open for the owner to confirm, that's fine too — no CI state will change the outcome.


八问(本轮结论):

  1. 是什么问题:作者请求在 rebase 后的新 head e23468ac 上重跑 CI,但该 PR 的改动(alignUsage 单遍游标 + 两个新文件)其实已经通过 batch7-wave7 合并(2221db1c9d,原始 head c8b86a3f)进入了 master。
  2. 什么原因导致的:批量合并流程把 PR 的原始 head 合进了 master,但 PR 本身保持 open;作者随后 rebase 出 e23468ac 时未注意到合并已发生。
  3. 影响范围与严重程度:对用户/行为零影响——master 上的内容就是已审核且 CI 全绿的 c8b86a3f;e23468ac 没有任何新内容(反而缺少 master 上同文件的 [PR] fix: keep whole-message inserts out of semantic appends #6738/[PR] fix: read logger attribution tags in cache usage #6742 新改动)。
  4. 怎么修复的:不重跑 CI(无东西可合,避免浪费饱和的 runner 池);向作者说明工作已随 batch7-wave7 上线,由作者/owner 关闭 PR。
  5. 是否有回归问题:无——本轮未改任何代码;master 上的 alignUsage 与已审核 head 逐字节一致。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:不涉及。
  8. 是否引入了新的配置:不涉及。

结论:PR 工作已在 master(f21b6018)上,无需重跑 CI;PR 可由作者/owner 关闭。

Louie0806 pushed a commit to Louie0806/billion-context that referenced this pull request Nov 14, 2026
@ranxianglei

Copy link
Copy Markdown
Owner

The fix from this PR is already in master (it landed via the wave-7 batch, now superseded by #6758 and in master from f21b601 onward). Closing as superseded — thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cache-diff usage alignment repeatedly scans the entire request history

2 participants