Skip to content

fix: close the four retrieval-delivery loss windows (#1343) - #1350

Closed
ranxianglei wants to merge 3 commits into
masterfrom
2026-09-25_retrieval-delivery-windows
Closed

ranxianglei wants to merge 3 commits into
masterfrom
2026-09-25_retrieval-delivery-windows

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Fixes #1343

What & why

The retrieve ack (retrieveToolName(ref) tool result) promises "the full text rides the next request" — but delivery rode volatile state with four loss windows where the ack was sent, the hit counted, and the text never arrived:

Window Where Loss
W1 restart/eviction persist.ts — pendingRetrievals never persisted (buildSession reset []) queue gone after restart
W2 post-drain forward failure drainPendingRetrievals spliced destructively in prepare*, then the upstream send could fail (network throw / 5xx) promised text silently eaten
W3 disarm storeEffectiveCcr — lane switches to a non-CCR wire → pendingRetrievals.length = 0 silently no trace
W4 unbounded float lane never drains (non-CCR wire after a mode switch / client stops re-requesting) queue grows forever

Fix

  • W1: the queue rides the persisted session file — buildRecord writes pendingRetrievals (only when non-empty; old files stay byte-compatible), buildSession restores it with a shape filter. Bounded by the in-memory cap, so the file bloat is bounded too.
  • W2: drainPendingRetrievals now also clears a transient requeue ticket (session.lastRetrievalDrain, never persisted); the two prepare* drain sites call trackRetrievalDrain after appending the batch to the outgoing list; forward()'s network-failure catch and the >= 500 branch call requeueRetrievalsOnFailure → batch goes back to the front of the queue, rides the next successful forward. Any new drain invalidates the ticket — a delivered batch can never be resurrected by a later request's error. Retries within forward (role-rewrite ladder, [bug] 假完成:模型把工具调用写成文字(无 tool_use 块)→ 轮次提前终止,agent 显示正常完成(#361 根因 follow-up) #371 backstop) re-send the same body, so they don't interact with the ticket.
  • W3: disarm clears loudly — [ccr] lane disarmed with N undelivered retrieval injection(s) (ack'd, never delivered) — dropping queue (#1343).
  • W4: queue capped at RETRIEVAL_QUEUE_MAX = 4; overflow drops the oldest with a log line (honest loss, same shape as disarm).

The stream-lane drains (non-stream tool rewrite, stream.ts) deliver inline in the tool result — ack + full text return atomically to the client, no failure window, no ticket needed (a fresh drain there still invalidates any stale ticket).

Tests

tests/ccr-retrieval-delivery.test.ts (new, 5):

  1. W2 — failed forward requeues the drained batch, order preserved, ticket consumed, second failure must not duplicate;
  2. W2 guard — a delivered batch is never resurrected (fresh drain invalidates the stale ticket);
  3. W3 — disarm clears the queue (and stays crash-free re-arming);
  4. W4 — 6 retrieves cap the queue at 4, oldest dropped;
  5. W1 — writeNow → loadSync disk round-trip restores the queue; empty queue writes undefined (byte-compatible files).

Existing store/ccr-plugin/ccr-v2 suites: 31/31 unaffected. Typecheck 0. Red-on-master: the new file fails to import the new API (1 fail) — behavior covered by the fix diffs above.

The retrieve ack promises the full text rides the next request, but
delivery rode volatile state with four loss windows:

- W1 restart: the queue was never persisted (persist.ts reset it to []);
  now it rides the session file (buildRecord/buildSession), bounded by
  the in-memory cap.
- W2 post-drain forward failure: drainPendingRetrievals spliced the
  queue destructively before the upstream send, so a network failure or
  5xx silently ate the promised text. drain now hands back a requeue
  ticket (session.lastRetrievalDrain, transient); forward's network-fail
  catch and the >=500 branch requeue the tracked batch onto the queue
  for the next successful forward. A fresh drain invalidates the ticket,
  so a delivered batch can never be resurrected by a later request's
  error.
- W3 disarm: storeEffectiveCcr cleared the queue silently; now logs the
  dropped count before clearing (ack'd, never delivered).
- W4 unbounded float: a lane that never drains (non-CCR wire, client
  stops re-requesting) accumulated full-text injections forever; the
  queue is capped at 4, oldest dropped with a log line.

Fixes #1343
@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-09-25_retrieval-delivery-windows (7df48f0)

Option A — Install from npm PR tag (recommended)

npm install -g billion-context@pr-1350

Each push to this PR publishes a new version under the pr-1350 npm tag.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr1350.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Reviewing PR #1350 (fixes #1343) now. Plan: (1) direction gate — check the implementation against the contract approved in #1343 floor 9 (delivered-at-completion, retry semantics, restore path, uniform coverage at all drain sites); (2) regression inventory on persist.ts / store.ts / server.ts touched paths; (3) code + test-effectiveness pass; (4) local typecheck/test/build on the rebased head. Will report with a verdict here shortly.

The W1 record line read session.pendingRetrievals.length unconditionally,
throwing TypeError on any Session object lacking the field (hand-built test
fixtures across ~20 suites; typecheck covers src/ only). Defensive read at
the serialization boundary; behavior unchanged for well-formed sessions.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Review of #1350 (fixes #1343) — reviewed against current master (d3c8af8), full local verification run, duplicate screening done. Verdict: mergeable pending two owner rulings below. One blocking defect found and already fixed on the branch (commit 5805acfb, pushed); CI is now fully green (11/11 checks).

① Direction gate (original issue)

#1343 is real and correctly layered: the ack promises "full text rides the next request" while delivery rode volatile pendingRetrievals — the fix attacks that producer (the delivery contract), not symptoms. The impl deviates from the floor-5/floor-9 plan in two documented ways (queue persisted directly instead of a metadata ledger; requeue ticket instead of drop-and-log) — both defensible simplifications, and W1-by-persistence is arguably stronger than ledger-rehydrate (no store-missing dependency). Two floor-9 contract points are unmet without being documented — see ③-B/C.

② Regression audit (existing behaviors touched)

Behavior Conclusion
buildRecord record shape Additive optional field; byte-compatible when empty (test 5 pins it). Was introducing a crash regression — fixed in 5805acfb (see ③-A)
buildSession restore Old files unaffected (field absent → []) ✓
drainPendingRetrievals Now invalidates stale ticket + markDirty → a session write now happens on every retrieve/drain. Intended for W1 durability; slightly more frequent writes, no semantic drift
storeEffectiveCcr disarm Clear semantics unchanged; now logs (W3, intended, disclosed in PR body)
executeRetrieve Behavior change (disclosed): old = unbounded queue → new = cap 4, oldest dropped + warn log. Also markDirty on retrieve
forward() network-catch / ≥500 branch New hooks early-return when no tracked batch → zero behavior change for non-CCR sessions. Internal retries (role ladder) re-send the same body, so the drained batch rides them — floor-9 point 1 satisfied
Wire shape / both compression modes Unchanged — injections ride the same channel as before; nothing newly injected. Proxy mode gets W1/W3/W4 store-level protections for free (incl. decompress-range injections)
Persistence version / update.ts / kernel pin / message-ref logic / security / version field All untouched ✓

③ Code findings

A. [BLOCKING — FIXED in 5805acfb] buildRecord crash → 54 test failures, all four CI test jobs red. The W1 line read session.pendingRetrievals.length unconditionally (src/persist.ts:679). Every hand-built session fixture across ~20 suites omits the field (they typecheck because tsconfig.build.json covers src/ only) → TypeError: Cannot read properties of undefined (reading 'length') in persist-zstd / encryption / stale-snapshot / #401 / export / #286 / issue404 / leaf-block suites. Verified A/B: master green (2646/2646), PR head red (2586/2651, 54 fail, identical stack at persist.ts:679). Fix: defensive read (session.pendingRetrievals?.length ?? 0) > 0 at the serialization boundary — consistent with drainPendingRetrievals' existing ?. style, zero behavior change for well-formed sessions. After fix: local typecheck 0, 2651/2651 pass, build OK; CI on 5805acfb: 11/11 success. The PR body's "existing suites unaffected" claim had only checked the 3 store/ccr suites — the full suite caught this.

B. [OWNER RULING NEEDED] Terminal 4xx after drain = silent loss. Requeue covers network-throw + status >= 500 only. If forward exhausts its retry ladder and returns a terminal 4xx (incl. 429 rate-limit, which is transient), the spliced batch is gone: ticket invalidated by the next request's drain, no log ties the loss to the retrieval promise. That violates acceptance criterion 1 ("every loss is logged"). Requeueing on 4xx is risky (if the payload itself triggered a 400, re-sending loops), so my recommendation: consume the ticket on terminal 4xx with a loud drop-and-log (refs + count + reason), and optionally treat 429 as the transient class worth requeuing. Alternatively, explicitly ratify terminal-4xx as accepted permanent loss in the PR body — but then it must at least log.

C. [OWNER RULING NEEDED] Floor-9 point 3 (corrective notes) absent. W3/W4 losses are logged but the model is never told "your retrieve was not delivered — re-issue", so "correctable" in criterion 1 currently means "recoverable from the store if the model remembers to look". The floor-9 contract said corrective notes ride the system-role ephemeral channel. My recommendation: defer to a follow-up issue — the core no-silent-loss property holds, the note is model-honesty polish, and this PR is already tight. Your call: defer (+issue) or implement here.

D. Floor-9 point 4 partially met. Stream-lane inline atomicity (stream.ts ×2, stream-google.ts) is documented ✓, but the proxy-mode internal-loop drain sites — src/loop/core.ts:706 and src/compress-loop-responses.ts:230 — have the same window-2 shape (drain → internal replay upstream call fails terminally → batch lost silently) and are neither ticketed nor documented-as-deferred. Ask: one explicit deferral sentence (PR body + code comment) or extend the ticket there. Related: decompress-shared.ts:296 range-restore pushes bypass the cap (bounded by blocks-per-call; they do benefit from W1 persistence).

E. Nits (non-blocking). ① Requeue unshift bypasses RETRIEVAL_QUEUE_MAX → transient max 8 entries (bounded, harmless). ② Restore filter checks only typeof m.text === "string" (not role/id) — a corrupted entry could pass; low risk. ③ Disarm log carries count+reason but no refs (criterion wording says refs+count+reason); overflow log names only the single dropped id. ④ resetSessionCompression (src/session.ts:509) still clears the queue silently — justified (refs re-issue on rebase, delivering would misattribute) but inconsistent with W3 loudness; a log line when non-empty would close the pattern.

④ Test quality

The 5 new tests have teeth: order preservation + no-duplicate-on-second-failure, delivered-never-resurrected guard, cap with oldest-dropped, and a real disk round-trip (writeNow→loadSync) plus the byte-compat assertion. Gaps: loudness isn't pinned by an assertion (log capture omitted — acceptable), there's no server-level test that a requeued batch actually rides the next forward (the server.ts wiring is unit-untested), and no proxy-mode window-2 test (floor-9 offered the documentation alternative — see D). The codex e2e suite passed on head, covering the real pipeline.

⑤ Diff cleanliness

Clean: 5 files, every line on-purpose, no lockfile/version churn, linear history. Duplicate screening: searched open+closed PRs/issues — this is the only work touching #1343 (#1345 and #1357 are distinct root causes). No parallel effort.


Bottom line: core delivery-safety verified end-to-end (local A/B + full suite + e2e green on 5805acfb). Two things need your ruling before merge: (B) terminal-4xx gap — add drop-and-log (my recommendation) or ratify as accepted loss; (C/D) floor-9 points 3 & 4 — defer with follow-up issues (my recommendation) or fold in now.

中文摘要:修复了 CCR retrieve 承诺文本的四个丢失窗口(队列随会话文件持久化、转发失败重排队、disarm 显式告警、队列上限 4);本轮 review 直接在分支上修掉了 buildRecord 对缺 pendingRetrievals 字段的会话崩溃(导致 54 个既有测试失败、四个 CI test job 全红),推回后本地 2651/2651 全绿、CI 11/11 通过;剩余两项待你拍板——终态 4xx(含 429)丢批是否补显式日志(建议补)、floor-9 第 3/4 点(纠正性提示、proxy-lane 两个 drain 站点)是延后立项还是本 PR 补齐(建议延后+立项)——核心安全属性已验证,拍板后即可合并。

A lane switch left the in-flight drain ticket armed: an already-delivered batch could be resurrected by an unrelated later forward failure once the lane re-armed (the exact stale-trailing-full-text case the disarm branch guards against). Clear lastRetrievalDrain on disarm; regression test proves red-on-parent.
@ranxianglei

Copy link
Copy Markdown
Owner Author

[bot] 🏷 关闭为已取代(superseded)— 已对 master c030b56 逐窗核验。

#1343 的四个丢失窗口全部由 #1367(merge 2a0eff72,ccr-retrieve-delivery-lifecycle)在 master 上关闭,其生命周期设计覆盖本 PR 的方案:

窗口 本 PR 方案 master 覆盖
W1 restart/eviction 持久化+恢复 pendingRetrievals durable ledger session.metadata.ccrUndelivered(随 session 文件 round-trip)+ 一次性 reconcileReloadedRetrievals()(ccrReconcilePending,persist.ts buildSession 置位;一次性旗标修复即 6d7c9c3)——丢失 ref 被大声 drop(reason "proxy restarted before delivery")+ 模型侧纠正注记
W2 post-drain forward failure requeue ticket(lastRetrievalDrain) plugin-lane prepare* 改非破坏性 snapshotPendingRetrievals,attachedRetrievals 挂在 Prepared 上 → 上游成功才 commitRetrievals,失败 dropRetrievals(...) 带日志;proxy-mode drain 即时提交(ack+injection 同一响应流,无跨请求窗口)
W3 silent disarm 大声日志行 dropRetrievals(session, allRefs, "CCR disarmed before delivery")(src/store.ts:37)——warn 日志含 refs+count+tokens、retrieveDropped 计数、纠正注记
W4 unbounded float 硬上限 RETRIEVAL_QUEUE_MAX=4 pruneExpiredRetrievals() TTL(默认 10min,BILI_CCR_RETRIEVAL_TTL_MS)——大声 drop,"no silent multi-turn leakage"

测试:#1367 内 tests/store.test.ts +107 行生命周期套件。本分支若合入将与 master 的双轨管理同一队列(磁盘队列 vs ledger+reconcile、requeue ticket vs snapshot/commit/drop、cap-4 vs TTL),状态形状直接冲突。

设计差异留档:master 选择 TTL 限界寿命而非硬大小上限——队列规模由 TTL 窗口内的 retrieve 活跃度界定而非常数;如日后想要硬上限是独立小改动,不属于本 PR。

Superseded by: #1367(merge 2a0eff7;reload-reconcile 旗标修复 6d7c9c3)。审计记录见 #1417。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(ccr): plugin-mode retrieve acknowledgement can outlive its volatile full-text delivery

1 participant