Skip to content

[PR] perf(persist): keep jsonWriter=off index builds out of the request path - #6986

Open
mariopasquali wants to merge 2 commits into
ranxianglei:masterfrom
mariopasquali:2026-11-17_writer-off-perf
Open

mariopasquali wants to merge 2 commits into
ranxianglei:masterfrom
mariopasquali:2026-11-17_writer-off-perf

Conversation

@mariopasquali

Copy link
Copy Markdown
Contributor

(ai: claude-opus-5-5)

Since #6480 pure SQLite (jsonWriter=off) is the default. Two places in that path run a full
index build synchronously when nothing needs it yet:

  1. scheduleSave arms the coalesced build and fires it from a microtask. During a request that microtask
    runs between the pipeline and the upstream forward, so every turn pays the build before the request
    leaves. Now, while the session is in flight, the armed build re-checks on the debounce interval and runs
    once the request has released the session. writeNow, flushSync and shutdown still take over an armed
    build, as before.
  2. flushSync (eviction) rebuilt and re-ingested the whole session even when the index already holds that
    exact revision (durableRevision === revisionEpoch, the same proof the idle spill uses). Now it skips
    the rebuild in that case and keeps the rest of the flush unchanged.

Measured on one real 38 MB session (writer off, same machine, 3 runs each):

master this PR
event loop blocked right after scheduleSave during a request 553–1,610 ms 3 ms
flushSync of a session the index already covers 483–952 ms 7–9 ms

On a larger session (~400 MB in memory) the second case blocked the event loop for about 55 s in our
install, delaying requests of other sessions.

Related: #6786 skips the redundant build when ingestRowsNow already wrote the same revision. This PR
is separate: it moves any remaining build after the request, and covers eviction.

Tests: tests/storage-writer-off-in-flight.test.ts (a save armed during a request does not build until
the request ends; an idle session still saves right away) and tests/storage-writer-off-clean-evict.test.ts
(an unchanged session is not rebuilt on flushSync; a newer revision is). Both fail on master.

Checks run: npm run typecheck, npm run check:unused-exports, storage + persist tests (414, 0 failures),
writer-off and durability-before-render tests (27/27). Full suite: 12,370 tests, 12,293 pass, 1 fail,
76 skipped. The failure is tests/issue2929-codex-fold-double-miss.test.ts ("scenario miss: no ephemeral
nudge in the pre-fold request"); it fails the same way on unchanged master 3e42aae and 301e1a5 on the
same machine now, and passed there earlier today, so it looks time-dependent and unrelated.
node tools/check-architecture-docs.mjs fails on master too (src/responses-system-sections.ts).

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重 (Duplicate screening): 不重复.

Related family (jsonWriter=off redundant index build) but DIFFERENT mechanism from #6786:

Different line ranges (no textual conflict), complementary, both reduce redundant jsonWriter=off builds. They should be sequenced (one rebased on the other) since they touch the same scheduleSave writer-off path.


8问 (Structured review checklist):

  1. 是什么问题: jsonWriter=off (pure SQLite, the default since [PR] storage: pure-SQLite default — JSON envelopes stop being written unless dual-write is requested (#6479) #6480) 路径上,两处在不必要时同步跑全量 index build:(a) scheduleSave 在请求中(inFlight>0)的 microtask 里触发 build,阻塞 event loop 553–1610 ms;(b) flushSync(eviction)对 index 已覆盖的 revision 仍然重建+重灌,阻塞 483–952 ms(大 session 可达 ~55 s)。

  2. 什么原因导致的: (a) scheduleSave 的 coalesced build 在 queueMicrotask 里无条件触发,不检查 session 是否 in-flight;(b) flushSync 的 writer-off 分支不检查 durableRevision 是否已覆盖当前 revision,总是重建。

  3. 影响范围与严重程度: 影响所有 jsonWriter=off session(默认配置)的请求路径。严重程度:性能问题(event loop 阻塞),非可用性缺陷。大 session(~400 MB in memory)的 eviction 路径阻塞 ~55 s 是主要痛点。

  4. 怎么修复的: (a) scheduleSave 的 writer-off trigger 改为 fire 函数:检查 inFlight > 0 时 defer(setTimeout re-check),inFlight === 0 时才 run;(b) flushSync 的 writer-off 分支加 covered 检查(durableRevision === revisionEpoch,与 idle spill 同一 disk proof),covered 时跳过 rebuild 但仍执行 tail fold + resident content drop。

  5. 是否有回归问题:

    • OUTPUT durability timing shift(主要行为变更): 模型的 response(OUTPUT)在 streaming 期间通过 markDirty 加入 session.lastMessages(inFlight>0)。OUTPUT 的 durability 依赖 scheduleSave build(pre-forward 的 ingestRowsNow 只保 INPUT)。[PR] perf(persist): keep jsonWriter=off index builds out of the request path #6986 之前:build 在 microtask 里触发(请求中),OUTPUT 在请求中 durable。[PR] perf(persist): keep jsonWriter=off index builds out of the request path #6986 之后:build defer 到请求结束(inFlight→0)+ ≤500 ms re-check delay,OUTPUT 在请求结束后 ≤500 ms durable。OUTPUT 的 crash window 从 ~microtask 增长到 ~请求结束+≤500 ms。PR body 没有显式说明这个 timing shift(它把 build 描述为 "redundant",但 OUTPUT build 不是 redundant)。需要 owner 确认这个 trade-off 可接受。
    • INPUT durability 不受影响(pre-forward ingestRowsNow 已保)。
    • shutdown 路径安全:drainWriterOffPending 直接 run 所有 armed build(绕过 inFlight 检查),force durability。
    • flushSync skip 安全:covered proof 与 idle spill 同一 gate(durableRevision === revisionEpoch);never-persisted session 永远不 covered(durableRevision 是 undefined,不是 0)。
  6. 是否破坏缓存: 不涉及(纯 persist/index build 路径,不涉及 prompt cache / KV cache)。

  7. 是否改了设计方向: 不涉及。两个改动都在现有 writer-off 路径内,不改架构/边界/数据流。

  8. 是否引入了新的配置: 不涉及。没有新的 config key / env var。


Detailed analysis (English):

Direction (①): Sound. Both changes target the jsonWriter=off redundant index build:

Code review (③): Clean. Follows existing patterns (token-check for coalescing, unref for the timer, same disk-proof gate as idle spill). Good why-comments explaining the rationale. No dead code, no magic numbers beyond the 50 ms floor.

Skipping maybeCutForkBase/noteMetaFp/noteDurable when covered is safe:

  • noteDurable is a no-op (durableRevision already === proofRevision).
  • maybeCutForkBase was already cut when the index was first built (it's called immediately before noteDurable in every build path: writeNow 3605-3607, scheduleSave run 3355-3365, flushSync).
  • noteMetaFp can't run (rec is null) and the metadata fingerprint is still valid (metadata unchanged ⇒ revision unchanged).

Subsumption still works: writeNow (3602) / flushSync (3909) / shutdown drainWriterOffPending (4107-4112) all delete/clear writerOffPending; fire checks the token (writerOffPending.get !== run) and no-ops. drainWriterOffPending runs run() directly (bypasses the inFlight check) — correct for shutdown (forces durability). The deferred setTimeout is unref'd (won't hold the process) and no-ops once the pending entry is deleted/cleared.

Tests (④): Both have teeth, both directions asserted:

  • storage-writer-off-in-flight.test.ts: T1 "armed during a request lands after release, not mid-request" (inFlight=1 → hasPending stays true + no index.db row after 120 ms; inFlight=0 → hasPending false + row present after 200 ms; debounceMs=20). T2 "idle session still saves right away" (inFlight=0 → build lands in microtask within 50 ms even with debounceMs=5000).
  • storage-writer-off-clean-evict.test.ts: T1 "flushSync of covered session does not rebuild" (monkeypatches UnifiedShadow.prototype.record to count; first flushSync builds (records>0), second flushSync of unchanged session → records unchanged, third after revisionEpoch++ → records grows). Uses tests/golden/storage-corpus/folded corpus.

CI status: 49 check runs on head f4ab941. TWO failures, BOTH unrelated to this PR:

  1. architecture-docs — PRE-EXISTING on master (PR body: "node tools/check-architecture-docs.mjs fails on master too (src/responses-system-sections.ts)").
  2. test (windows-latest, 22, 4) — not ok 910 - #2929: nudge + proxy-tool-round fold keeps re-request/next-turn byte-stable at tests/issue2929-codex-fold-double-miss.test.ts:232:16, error "scenario miss: no ephemeral nudge in the pre-fold request (inputs[6])". SAME test the PR body flagged as time-dependent and failing on unchanged master (3e42aae, 301e1a5). It's a NUDGE/FOLD test, NOT related to the persist/index-build changes.
    All other test shards success; all static gates success.

Diff cleanliness (⑤): Clean. 3 files, all on-topic (src/persist.ts + 2 new test files). No unrelated changes, no mass whitespace/reformat.

Verdict: Mergeable after owner confirms the OUTPUT-durability timing trade-off is acceptable (see 8问 #5). Both CI failures are pre-existing/unrelated. No defect requiring fix found.

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Fixed both CI failures on the PR branch. Pushed d55b5a971f to 2026-11-17_writer-off-perf.

What was red:

  1. architecture-docs — reference/architecture.md was missing src/responses-system-sections.ts ([Bug] Inconsistent system prompt separator in Responses internal re-request breaks cache prefix | [Bug] Responses 内部重请求的系统提示分隔符不一致,破坏缓存前缀 #6922, landed on master via the [Bug] Inconsistent system prompt separator in Responses internal re-request breaks cache prefix | [Bug] Responses 内部重请求的系统提示分隔符不一致,破坏缓存前缀 #6922 merge but the architecture map was never updated). The guard checks the merge commit, so the PR's merge tree had the file but the map didn't list it.

  2. test (3 shards) — all three failing shards ran the same test: #2929: nudge + proxy-tool-round fold keeps re-request/next-turn byte-stable (tests/issue2929-codex-fold-double-miss.test.ts). The test's PROXY_TOOL_AT/FOLD_AT anchors were calibrated for the nudge schedule {4,6,8,9} that existed while the Compression conflict banner (bili v0.1.205, node v24.19.0, win32-x64) #4521 armed-but-silent status line was present. Per-turn '[ACP status: …]' line keeps ordering the model to fold leftover small ranges right after a big fold — the opposite of what the compress receipt says #6928 (commit 4ccccab15f, already in this PR's base) removed that status line, which shifted the nudge schedule back to {3,5,12}. The nudge no longer lands at index 6, so the scenario-materialization assert fails.

Fixes:

  1. Added src/responses-system-sections.ts to reference/architecture.md at the top-level src/ section (next to responses-ws.ts). node tools/check-architecture-docs.mjs passes locally.

  2. Re-calibrated PROXY_TOOL_AT from 6→5 and FOLD_AT from 7→6 to track the new nudge schedule. Updated the comment block to document the Per-turn '[ACP status: …]' line keeps ordering the model to fold leftover small ranges right after a big fold — the opposite of what the compress receipt says #6928 schedule shift. Test passes locally.

八问:

  1. 是什么问题: Two independent CI failures — a stale architecture map entry and a test probe whose nudge-index anchors drifted after an unrelated kernel change removed the Compression conflict banner (bili v0.1.205, node v24.19.0, win32-x64) #4521 status line.
  2. 什么原因导致的: (a) [Bug] Inconsistent system prompt separator in Responses internal re-request breaks cache prefix | [Bug] Responses 内部重请求的系统提示分隔符不一致,破坏缓存前缀 #6922 added src/responses-system-sections.ts to master without updating reference/architecture.md; the architecture-docs guard only trips when the file appears in the merge tree. (b) Per-turn '[ACP status: …]' line keeps ordering the model to fold leftover small ranges right after a big fold — the opposite of what the compress receipt says #6928 removed the Compression conflict banner (bili v0.1.205, node v24.19.0, win32-x64) #4521 status line, which had been feeding the nudge gate's shown-marker and shifting the injection schedule; the codex 命中率问题分析 #2929 probe's hardcoded indices were calibrated for the pre-removal schedule.
  3. 影响范围与严重程度: CI-only — no runtime behavior change. The test still covers the same codex 命中率问题分析 #2929 invariant (post-fold re-request / next-turn byte parity); only the request indices moved. The architecture map fix is a docs-only addition.
  4. 怎么修复的: (a) Added the missing file entry to the architecture map. (b) Re-calibrated the probe's PROXY_TOOL_AT/FOLD_AT constants to the new nudge schedule and updated the explanatory comment.
  5. 是否有回归问题: No. The test's core assertions (summary anchor parity, byte-identity through the anchor) are unchanged. The architecture map fix adds one line; no existing entries were modified.
  6. 是否破坏缓存: No. No cache, retry, or incremental-cursor logic touched.
  7. 是否改了设计方向: No.
  8. 是否引入了新的配置: No.

CI is running on the new head (d55b5a971f); the mirror cache still shows the old SHA and will refresh within ~60s.

…eases the session

The coalesced index-only build is synchronous and costs seconds on a large
session; firing it from a microtask put it between the pipeline and the upstream
forward on every turn. While the session is in flight the armed build re-checks on
the debounce interval and lands once released; writeNow/flushSync/shutdown still
subsume it.
…already covers the revision

Eviction calls flushSync; in index-only mode it rebuilt and re-ingested the
whole session even when the durable revision proof (ranxianglei#4788 D2) already equals the
live revisionEpoch. On a 400 MB-weight session that stalled the event loop for
~55 s on the next request of another session.
@ranxianglei
ranxianglei force-pushed the 2026-11-17_writer-off-perf branch from d55b5a9 to afe430a Compare November 17, 2026 13:24
@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

Verdict: no merge conflict exists and CI is green — nothing to fix. No commit/push/force-push performed.

The wake premise ("merge state: dirty" / conflict) does not hold up against the live sources checked this run:

1. No merge conflict (verified three independent ways)

  • GitHub API: mergeable: true.
  • Empirical: git merge --no-commit --no-ff upstream/master → Already up to date.
  • Relationship: current master (77fe0b1d63) is exactly the PR's base; the PR head afe430afa1 is 2 commits ahead of master, master is 0 ahead. There is nothing to rebase or merge — the branch already contains master's tip.

2. CI is green (verified by a single authoritative read)

  • The mirror upstream-check cache was stale (fetched_at 13:28 UTC, ~18 min old) and still showed 13 checks in_progress. A single GitHub read of the head commit's check-runs shows all 50 completed: 37 success + 13 skipped, 0 failed, 0 pending.

3. The real (and only) blocker: human review

  • mergeable_state: blocked with 0 reviews submitted — branch protection is waiting on an approving review. That is human-only; it is neither a conflict nor red CI, and it is not mine to satisfy.

Action taken: none. Local branch pr6986-rebase already equals the PR head (afe430afa1); there was no conflict to resolve and no failing CI to fix, so I made no commit, push, or force-push.

Observation (out of scope for this lane): the mirror upstream-check cache was not refreshing (~18 min stale), which is why the wake saw a phantom "dirty". Not fixable from here; flagging for the owner.


八问(中文):

  1. 是什么问题:系统唤醒称 PR「与 base 冲突(merge state: dirty)」,要求 rebase/merge 解冲突。查证后实际不存在冲突。
  2. 什么原因导致的:镜像 upstream-check 缓存过期(fetched_at 13:28 UTC,约 18 分钟未刷新),其「dirty / 13 pending」是旧快照;真实状态是 mergeable=true 且 CI 全绿。冲突为误报。
  3. 影响范围与严重程度:无实际影响——PR 本身干净,只是被误报成冲突。严重程度:无(纯误报,未破坏任何东西)。
  4. 怎么修复的:无需修复。三重实证:①GitHub API mergeable=true;②本地 git merge --no-commit --no-ff upstream/master = "Already up to date";③当前 master(77fe0b1d63)= PR base,PR 领先 2 commit、master 落后 0,没有可 rebase/merge 的内容。CI 用单次权威读取确认 50 项 check-run 全 completed(37 success + 13 skipped,0 failed)。未做任何 commit / push / force-push。
  5. 是否有回归问题:无——本轮未改动任何代码。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:不涉及。
  8. 是否引入了新的配置:不涉及。

收工判定:PR 干净、无冲突、CI 全绿,唯一剩余阻塞是人工 review(0 reviews,branch protection)——非冲突、非红 CI,不在本 CI/冲突车道职责内。无可修项,收工。

@mariopasquali

Copy link
Copy Markdown
Contributor Author

(ai: claude-opus-5-5)

On the output-durability question in the review: yes, that is the trade-off, and it is bounded.

  • Before: the build for the answer ran in a microtask while the request still held the session, so it
    blocked the event loop for 553–1,610 ms on the bench sessions before the response finished.
  • After: it runs once the request releases the session, re-checked every max(50, debounceMs) ms. With the
    default debounceMs of 500 the answer is in the index at most about 500 ms after the request ends.
  • That is the same window the legacy JSON writer has always had (it is debounced by the same 500 ms).
  • The input is still saved before forwarding, and shutdown, writeNow and eviction still run an armed
    build right away, so a clean stop or eviction does not lose the answer. Only a crash inside that
    window does, and the client's next request carries that answer in its history again.

If the window should be shorter, the build could start directly when the last in-flight request releases
the session instead of waiting for the next re-check. That would be a small follow-up commit here.

@ranxianglei

Copy link
Copy Markdown
Owner

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

🔶 需要决策

The author clarified the output-durability trade-off (answering the review question) and offered a follow-up commit to shorten the window. The CI/conflict side is clean — no merge conflict, CI green, nothing for this lane to fix. What's open is a design decision for the owner: whether to shorten the answer-durability window.

The decision — how long an answer can stay out of the index after its request ends:

  • Option A — keep as-is (author's recommendation). The build runs once the request releases the session, re-checked every max(50, debounceMs) ms (default debounceMs=500), so the answer lands in the index at most ~500 ms after the request ends. That is the same window the legacy JSON writer always had (debounced by the same 500 ms). The input is still saved before forwarding; shutdown / writeNow / eviction run an armed build immediately; only a crash inside that ~500 ms window loses the answer, and the client's next request re-sends it in its history.
  • Option B — follow-up commit (shorter window). Start the build directly when the last in-flight request releases the session, instead of waiting for the next re-check. Cuts the window toward 0 at the cost of a small extra code path.

Recommendation: Option A (keep as-is). The trade-off is bounded and matches the legacy writer's durability; the only loss is a crash inside a ~500 ms window, and the client re-sends the answer. Option B buys a marginal durability gain for added complexity.

Default if undecided: Option A (no follow-up commit).


八问(中文):

  1. 是什么问题:作者就 review 里的「输出持久性」问题澄清了取舍,并主动提出一个可选的 follow-up commit(把答案进入索引的窗口从 ~500ms 缩到 ~0)。这是设计决策,不是 CI/冲突问题。
  2. 什么原因导致的:本 PR 把 jsonWriter=off 的索引构建从「请求持锁期间的 microtask」移到「请求释放会话后、每 max(50, debounceMs) ms 复检」,以消除事件循环阻塞(基准会话 553–1,610 ms)。代价:答案最晚晚 ~500ms 进索引(与旧 JSON writer 的 500ms debounce 窗口相同)。
  3. 影响范围与严重程度:仅 jsonWriter=off(纯 SQLite,默认)路径;仅「窗口内崩溃」丢答案,客户端下次请求会重带。严重度:持久性边缘,非可用性缺陷。
  4. 怎么修复的:本轮无代码改动——CI/冲突侧干净(无冲突、CI 全绿),设计取舍留给 owner 拍板。
  5. 是否有回归问题:不涉及(未改代码);作者论证该窗口与旧 writer 一致,无新增回归。
  6. 是否破坏缓存:不涉及。
  7. 是否改了设计方向:不涉及(这是 PR 本身的设计,非我改动)。
  8. 是否引入了新的配置:不涉及(debounceMs 为既有默认 500)。

判定:CI/冲突车道无可修项;follow-up commit(缩短窗口)是 owner 的设计决策,建议 Option A(保持现状),未决则默认 Option A。已打 needs-decision 标签。

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants