Skip to content

[PR] perf: scan cache body boundaries in linear time - #6843

Closed
Afloat16 wants to merge 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-15_linear-body-stability
Closed

Afloat16 wants to merge 1 commit into
ranxianglei:masterfrom
Afloat16:2026-11-15_linear-body-stability

Conversation

@Afloat16

Copy link
Copy Markdown
Contributor

Fixes #6839

Problem

Settling a normal appended turn can block the proxy event loop for seconds once the retained transcript reaches a few MiB. messagesArrayCloseOffset performs separate Buffer.indexOf searches for quotes, backslashes and structural delimiters from every segment. Searches that find no delimiter, or a distant array boundary, repeatedly traverse the same remaining body.

Change

Scan each byte once while retaining the existing string, escape-parity and UTF-8 byte-offset rules. Run the array-boundary and semantic-insertion checks only after the cheap append/head decisions need them. Both compression modes share this bookkeeping; request bodies, compression behavior and diagnostic classifications are unchanged.

Add a regression through the public noteForwardedBody / settleUsageReport path. It measures native search spans without a wall-clock assertion: unchanged source searches 793,522,314 bytes for a 219,319-byte transcript and fails the bound. Additional controls cover all three history keys, nested arrays, escaped delimiters, CJK text, actual mid-history edits and an unlocatable array.

Validation

  • New tests on unchanged master: 1 failure, 4 passing controls.
  • Candidate: 88 targeted tests passed, including existing UTF-8, tail/mid insertion, cache capture and ledger suites. The five new cases were rerun after the final test type-signature adjustment.
  • Full-project TypeScript check, npm run build, test-import-graph gate and git diff --check: passed.
  • Fresh upstream master was rechecked at 2e3cfc973b8620b1f7bc8a415ddcbd31e7f4aaee before publishing. Remote tree 19a3d90dd77c9131e31c14deaac07294bb8b424c exactly matches the tested local index.
  • Full regression remains the hosted CI verdict on this head.

Local benchmark

Node v24.22.1, Linux; alternating user/assistant prose, followed by one appended user message. Capture and settling are included; body construction is outside the interval. One warmup and three measured iterations, each checking the existing tail classification:

Messages Body bytes Baseline median Candidate median
1,000 431,519 44.46 ms 4.53 ms
4,000 1,729,019 1,307.23 ms 13.69 ms
8,000 3,459,019 10,636.32 ms 32.36 ms

Reproduce with node --import ./tests/preload-test-env.mjs --import tsx scripts/bench-cache-array-scan.mjs 1000 4000 8000. Timings vary with machine/load; these are local bookkeeping measurements, not end-to-end proxy throughput claims. CI uses the deterministic work-bound test instead of timing thresholds.

Prior work

Checked all 134 Afloat16 PRs plus all-state body-stability/suffix-scan searches and relevant commit history. #2139, #2870, #2983 and #3479 establish the retained classifications; #6722 optimizes ordered usage alignment and is already on master. This change addresses the remaining boundary scanner.

Avoid repeated suffix searches when settling a long transcript and defer
boundary work until append/head checks need it.

Fixes ranxianglei#6839
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Reviewed PR #6843 against current master (f648217c). This is a well-scoped, correct, and well-tested fix. I verified the regression coverage independently and ran the related suites rather than trusting the description.

What I verified (evidence, not "should work")

  • The regression test is real. I ran the new tests/cache-body-boundary-scan.test.ts on BOTH master and the PR head:
    • master: 1 failure, 4 passing controls — the work-bound case fails with searched 793522314 bytes for a 219319-byte body (≈3618×, violating the 8× bound). This exactly matches the PR's claim.
    • PR head: all 5 pass.
  • No classification regression. The 4 control cases (three history keys × nested/escaped/CJK content, plus the unlocatable-array case) pass on BOTH master and PR head — proving the append/head/tail/mid classes are byte-identical before/after.
  • Related suites green on PR head: 89 tests (UTF-8 byteLcp, tail-class, insertion-boundaries, unicode, capture-pairing, body-stability-visibility, seam-capped-attribution, seam-detector) + 51 tests (cache-ledger main, seam-fuzz, seam-client-insert) — all pass.
  • Typecheck clean on the PR head. (My sandbox was missing the optional @deepseek-ai/cordis dev dep; the only tsc errors were in an unrelated, PR-untouched test file and disappeared once it was installed.)
  • CI: all completed lanes green (gate ×5, unused-exports, test-import-graph, kernel-guard, architecture-docs, …); 6 secondary guard/docs checks are queued (not failed). The PR is 4 commits behind master but mergeable (no conflicts; the behind commits are gate/docs and don't touch the PR's files).

Why the fix is correct (independent read of the diff)

  • messagesArrayCloseOffset (src/cache-ledger.ts:988): the state machine (inStr/depth/want/keyAt, backslash-run escape parity, UTF-8 byte-offset safety) is unchanged. Only the fast-forward strategy changed — from Buffer.indexOf jumps (which rescan the entire remaining suffix when a delimiter is absent or distant → quadratic in segment count) to a one-byte-per-iteration step (true linear). The function's own doc comment already promised "one linear pass keeps multi-MiB bodies cheap"; the old code didn't deliver it, the new code does.
  • settleUsageReport (src/cache-ledger.ts:~2005): the eager ternary was restructured into if / else if / else so messagesArrayCloseOffset and isMidInsertionAppend (both pure) are computed only in the would-be-mid/tail branch. The bs value is identical for every input — a pure lazy-eval of the same condition chain. Bonus: the common append/head cases now skip the scan entirely.

Minor (non-blocking)

  • The PR is behind master (4 commits, gate/docs only) — a rebase would be tidy but isn't required.
  • The 8 in the test's scannedBytes <= previous.length * 8 bound is a bare literal; a named constant would be marginally clearer, but it's a one-off test bound with a self-explanatory assert message — not worth a rework.

结构化八问

  1. 是什么问题:长会话(数 MiB 保留 transcript)每追加一条普通消息,settleUsageReport 里的 body-stability 分类就会把代理事件循环阻塞数秒(PR 本地基准:3.46 MB body 基线中位 10.6 s),期间该进程上的全部请求都被卡住。
  2. 什么原因导致的:messagesArrayCloseOffset 对每个字符串/filler 段都从当前位置发起多次 Buffer.indexOf(引号/反斜杠/结构分隔符)。正文通常没有反斜杠、数组边界又很远,每次搜索都扫到 body 末尾 → 段数 × body 长度 = 二次方;且该 helper 在廉价的 append/head 分类判定之前就被急切求值。
  3. 影响范围与严重程度:任何保留 transcript 达数 MiB(8 MiB BODY_FULL_CAP 以内)的长会话,每次普通追加都触发;严重度为事件循环阻塞数秒(拖慢该进程全部流量),但分类结果本身不变——是性能/可用性问题,不是正确性或数据丢失。
  4. 怎么修复的:(a) 扫描器改为真正的单遍线性(每字节一步,状态机不变),兑现函数注释里"one linear pass"的本意;(b) 把扫描延迟到 would-be-mid/tail 分支才计算,常见 append/head 直接跳过。
  5. 是否有回归问题:无。单遍扫描与多 indexOf 版本行为完全等价(状态机/转义奇偶/UTF-8 安全均不变);分类重构是两个纯函数的惰性求值,bs 取值逐输入相同。89+51 相关测试在 PR head 全绿,4 个分类对照在 master 与 PR head 均通过(证明分类未变),typecheck 干净。高危模式逐项:无修 A 坏 B(唯一调用方就是该分类)、无 fail-fast 变吞错、阈值(64/1024)与 bs 输出格式无漂移。
  6. 是否破坏缓存:不涉及。这是诊断性记账(seam 取证用的 body-stability 分类),不触碰 cache key、缓存内容、重试、游标或去重键。
  7. 是否改了设计方向:不涉及。恢复文档已声明的线性本意,无架构/边界/数据流转向。
  8. 是否引入了新的配置:不涉及。无新 env/config/默认值。

判定:可合并。 范围清晰、修法打到根因(线性化 + 惰性求值,不是堵症状)、回归测试有牙(在 master 上真实失败、PR head 通过)、对照充分。CI 测试车道全绿,仅剩 6 个次要 guard/docs 检查在排队(非失败)。

@ranxianglei

Copy link
Copy Markdown
Owner

This fix is included verbatim (original commits and authorship preserved) in the merged batch PR #7074 — the changes are already on master. Closing to avoid a duplicate merge. 该修复已随批量 PR #7074 原样合入 master,为避免重复合并,关闭此 PR。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cache body-stability scans repeatedly traverse long transcript suffixes

2 participants