Skip to content

fix(ccr): give plugin-mode retrieve delivery a lifecycle so acks cannot outlive lost text - #1367

Merged
ranxianglei merged 4 commits into
masterfrom
2026-09-26_ccr-retrieve-delivery-lifecycle
Sep 26, 2026
Merged

ranxianglei merged 4 commits into
masterfrom
2026-09-26_ccr-retrieve-delivery-lifecycle

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Sep 25, 2026 •

Copy link
Copy Markdown
Owner

Summary

In plugin mode, acp_retrieve committed its success (ack text + retrieveHits) before the full text had any guaranteed delivery path: the ack rode the plugin tool response immediately while the full text sat in the volatile in-memory session.pendingRetrievals queue, appended only to a later upstream request. A restart/eviction, an upstream failure after drain, a CCR disarm, or a long run of non-qualifying requests could each leave the ack on record while the text was silently lost — or surfaced detached from context turns later.

This gives queued retrievals a delivery lifecycle so no ack can outlive a silently-lost text.

What changed

  • Lifecycle (src/store.ts): queued → attached → delivered | dropped. executeRetrieve now queues with a durable undelivered ledger in session.metadata.ccrUndelivered (round-trips to disk; no on-disk schema change). New functions: queueRetrieval, snapshotPendingRetrievals, commitRetrievals, dropRetrievals, pruneExpiredRetrievals, reconcileReloadedRetrievals, flushRetrievalNotes.
  • Plugin lane (src/server.ts): prepares now SNAPSHOT (not splice) the queue onto the outgoing body (Prepared.attachedRetrievals); forward() settles it at the final response — committed on 2xx, dropped-and-logged on network failure / non-2xx / transform failure. Closes all four windows from the issue.
  • Observable losses: every drop logs refs+count+reason at warn, bumps stats.retrieveDropped, and queues a corrective note flushed as an ephemeral trailing user message next round. Disarm terminates pending deliveries observably (no silent truncate, no stale flush on re-arm — preserves feat: enable CCR (acp_retrieve) on host/plugin lanes (#1271) #1273). Reload reconciles acked-but-never-delivered refs. A TTL expires loudly.
  • Two-riders: range-restore injections (feat: CCR v2 — covered-ref spans, range-level decompress, ID-returning search_context; CCR default-on (#1179) #1207, src/decompress-shared.ts) share the carrier but are ccr:false — removed at the delivery outcome but exempt from the CCR ledger/TTL/counters/note, preserving their original ride-along semantics.
  • Proxy mode unchanged: drainPendingRetrievals stays atomic in-flow (four-wire happy path + pair integrity intact); it now also commits delivery so proxy retrieves count in retrieveDelivered.
  • Counters (src/session.ts, src/persist.ts): additive stats.retrieveDropped / stats.retrieveDelivered; restored on load.

Behavior changes (old → new)

Area Old New
CCR disarm silent truncate of pendingRetrievals each acked ref logged (refs/count/reason) + corrective note; still removed (no stale flush on re-arm)
Post-drain upstream failure injection destroyed silently dropped-and-logged + corrective note
Restart / eviction acknowledged retrieve silently lost detected on reload, logged + corrective note
Long gap / no qualifying request injection could surface turns later, unlabeled TTL expiry → loud drop + note (default 10 min)
Health metric retrieveHits counted acked (incl. undelivered) retrieveHits unchanged; new retrieveDelivered counts confirmed deliveries, retrieveDropped counts losses
New env var — BILI_CCR_RETRIEVAL_TTL_MS (default 600000 ms; 0 disables expiry)

The agent-facing ack is unchanged (still synchronous, same text); only the bookkeeping around it becomes honest.

Known bounded residuals (accepted at review)

  • Plugin-lane settlement point: delivered settles at the upstream 2xx headers; a stream breaking mid-body after headers counts as delivered with no corrective note (rare; clean-finish settlement is a follow-up candidate).
  • Corrective-note channel: notes ride a trailing ephemeral user message (same channel as imageFullTrailingNote, Feature: image pre-compression for screenshot-heavy tool results (resize/quality routing) #1095), not a system-role ephemeral channel.
  • Proxy lane: commits at drain time; a round failing after drain loses the batch while counting it delivered (wire unchanged; metrics-only honesty gap).
  • Reconcile scope: ccrReconcilePending is consumed by plugin-lane prepare only; a restarted session that stays on the proxy lane keeps its stale ledger rows in metadata until it next touches the plugin lane (model-invisible, bookkeeping only).
  • Concurrent same-session requests: both may attach and deliver the same refs (model sees the full text twice — token cost, not loss; ledger outcomes are idempotent).

Pre-flight

  • npm run typecheck — clean
  • npm test — 2628 pass, 0 fail, 10 skipped (new tests cover all four windows + two-riders isolation + proxy regression)
  • npm run build — success

Closes #1343

…t outlive lost text

In plugin mode, acp_retrieve committed its success (ack text + retrieveHits)
before the full text had any guaranteed delivery path: the ack rode the tool
response immediately while the full text sat in the volatile in-memory
session.pendingRetrievals queue, appended only to a LATER upstream request. A
restart, an upstream failure after drain, a CCR disarm, or a long run of
non-qualifying requests could each leave the ack on record while the text was
silently lost or surfaced detached from context.

Give queued retrievals a lifecycle (queued -> attached -> delivered | dropped):
- executeRetrieve queues with a durable undelivered ledger in session.metadata
  (round-trips to disk; no on-disk schema change).
- Plugin-lane prepares SNAPSHOT (not splice) the queue onto the outgoing body;
  forward() settles it at the final response: committed on 2xx, dropped-and-
  logged on network failure / non-2xx / transform failure.
- Disarm terminates pending deliveries observably (no silent truncate, no stale
  flush on re-arm); reload reconciles acked-but-never-delivered refs; a TTL
  (BILI_CCR_RETRIEVAL_TTL_MS, default 10min) expires loudly.
- Every loss logs refs+count+reason, bumps stats.retrieveDropped, and queues a
  corrective note flushed as an ephemeral trailing user message next round.
- New additive counter stats.retrieveDelivered counts confirmed deliveries.
  retrieveHits semantics are unchanged.

Range-restore riders (#1207) share the carrier but are ccr:false: removed at
the delivery outcome but exempt from the CCR ledger/TTL/counters/note, so their
original ride-along semantics are preserved. Proxy-mode drain is unchanged
(atomic in-flow) -- four-wire happy path and pair integrity intact.

Tests cover all four windows plus two-riders isolation and the proxy regression.
@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-09-26_ccr-retrieve-delivery-lifecycle (d8bb165)

Option A — Install from npm PR tag (recommended)

npm install -g billion-context@pr-1367

Each push to this PR publishes a new version under the pr-1367 npm tag.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr1367.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

收到。这是 agent 已完成的 PR,我不重做——现在按 review 规范走:拉分支核对代码 → 回归审查(既有行为清点)→ 代码/测试有效性审查 → CI 状态确认,结论稍后贴在本线程。

ework-agent added 3 commits September 26, 2026 03:26
… clears restored first)

reconcileReloadedRetrievals gated on session.restored, but getSession()
clears restored on the first request touch — BEFORE prepare runs the
reconcile in that same request. The window-1 restart detection (#1343,
the headline scenario of this PR) therefore never fired in production:
acked-but-never-delivered ledger entries survived silently with no log,
no counter, no corrective note. The unit test masked this by setting
restored=true manually.

Fix: persist's buildSession now arms ccrReconcilePending (in-memory
one-shot); reconcile keys off it and consumes it. resetSessionCompression
clears it alongside the ledger. W1/W1b updated to pin the production
ordering (flag armed, restored already cleared).
…TORE line

floor-9 polish item: the new delivery/drop counters were not surfaced
anywhere user-visible. STORE line gains '· delivered N' / '· dropped N'
segments when non-zero.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 PR #1367 review complete. Verdict: not yet mergeable — one blocker found & fixed on the branch; three contract gaps need your call.

① Direction (原 issue): sound. The lifecycle (queued→attached→delivered|dropped) fixes each window at its decision point rather than masking symptoms: snapshot-not-splice makes internal retries re-attach automatically, settlement is exactly-once per client request, every drop is observable (warn log + counter + corrective note). Root cause addressed, not papered over.

② Regression audit (existing behaviors touched):

  • Proxy-mode drain: wire behavior unchanged (injection still spliced atomically into the same request; pair integrity pinned by test). New: proxy retrieves now count in retrieveDelivered — metric-only, disclosed in the PR table ✓
  • CCR disarm (feat: enable CCR (acp_retrieve) on host/plugin lanes (#1271) #1273): still removes everything with no stale flush on re-arm; now observable instead of silent — disclosed ✓
  • Range-restore riders (feat: CCR v2 — covered-ref spans, range-level decompress, ID-returning search_context; CCR default-on (#1179) #1207): ccr:false exemption preserves ride-along semantics (removed at outcome, exempt from ledger/TTL/counters/note) — pinned by test ✓
  • Persistence: metadata-only round-trip via the existing spread, no schema change, backward-compatible with old files ✓; resetSessionCompression clears all three structures ✓
  • Note flush is ungated by ccrEnabled → notes buffer through disarm and flush on the next request (no stuck notes) ✓
  • Concurrency: prepare+forward run under the per-session lock → double-snapshot impossible. One benign race remains (⚪ below).

③ Code review:

  • 🔴 P1 (FIXED on branch, commit 6d7c9c3d): window-1 reload reconcile was dead code in production. reconcileReloadedRetrievals gated on session.restored === true, but getSession() (src/session.ts:314) clears restored on the first request touch — before prepare runs the reconcile in that same request (src/server.ts:1713 vs :2089). So after a restart, acked-but-never-delivered ledger entries were never logged, counted, or noted — exactly the silent loss bug(ccr): plugin-mode retrieve acknowledgement can outlive its volatile full-text delivery #1343 targets, i.e. the headline scenario of this PR. Test W1 masked it by setting restored = true manually. Fix: persist's buildSession arms a one-shot ccrReconcilePending; reconcile keys off it and consumes it; resetSessionCompression clears it. Both load paths (boot loadAll, LRU-miss loadSync) route through buildSession, so eviction-reload re-arms too. W1/W1b updated to pin the production ordering (flag armed, restored already false).
  • 🟡 P2 (your call): "delivered" = 2xx headers, not clean-finish. Floor-9 point 1 required delivered only when the response fully completed; the PR settles at header time (src/server.ts:4727). A stream breaking mid-body after headers counts as delivered with no corrective note. Rare trigger; stream-end settlement would touch the stream layer across all wires (bigger change). Options: (a) accept + document as bounded residual, (b) follow-up. The PR body currently reads as if window 2 is fully closed — needs a sentence either way.
  • 🟡 P3 (your call): corrective notes ride a trailing USER message, not the system-role ephemeral channel floor-9 point 3 specified. Practical risk low — imageFullTrailingNote (Feature: image pre-compression for screenshot-heavy tool results (resize/quality routing) #1095) uses the identical channel in the same code path — but it deviates from the approved wording; accept explicitly or fix.
  • 🟡 P4 (minor): proxy lane commits at drain time, so a proxy-mode round failing after drain loses the batch while counting it delivered (metrics honesty; wire unchanged). Floor-9 point 4 wanted uniform lifecycle OR documented deferral — please document or defer explicitly.
  • ⚪ Benign race (non-blocking): disarm (storeEffectiveCcr, src/server.ts:1857) runs outside the session lock, so a lane-switch/config-reload landing between snapshot and response can drop a batch that WAS delivered → false dropped count + false note. Self-healing (re-fetch works), rare. Optional hardening for a future pass.
  • ✅ Polished (commits 667728bf, d8bb1654): retrieveDelivered/retrieveDropped now surface on the acp_status STORE line (floor-9 polish item); BILI_CCR_RETRIEVAL_TTL_MS documented in CONFIGURATION.md.

④ Tests: real teeth (counter values, ledger state, note text, pair-integrity regexes). The single gap was W1 masking the P1 ordering bug — now pinned.

⑤ Diff: clean — one original commit, 6 files, all in scope, no version bump, no lock churn. My three review-fix commits are clearly labeled above it.

Evidence: CI fully green on original head 36515dab (11 checks incl. e2e + real-client lanes). On new head d8bb1654: local typecheck clean, full suite 2628 pass / 0 fail / 10 skipped, build success; GitHub CI re-running on the push.

Before merge: (1) your calls on P2/P3/P4 (accept-with-docs or fix); (2) rebase onto current master — the branch is 9 commits behind (incl. release v0.1.153); trial merge-tree against live master is conflict-free so the rebase should be trivial, then re-run gates. Happy to do the rebase once the calls land.

中文摘要:审查了 CCR retrieve 投递生命周期 PR——发现并已在分支上直接修复核心缺陷「重启对账因 getSession 先行清除 restored 标志而永不触发」(改为一臂 ccrReconcilePending 一次性标志并更新测试钉住生产时序),补齐了 acp_status 展示与 TTL 环境变量文档;剩余三项(以 2xx 头而非流结束记 delivered、纠正注走 user 消息通道、proxy 模式在 drain 时即记 delivered)均偏离 floor-9 已批契约,需 owner 拍板接受留痕或修复;待拍板 + rebase 最新 master 后可合并。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 CI follow-up: all 11 checks green on new head d8bb1654 (e2e, hermetic registry e2e, test ubuntu 22/24 + windows 22/24, version-guard, real codex + real pi lanes, build-artifact, codex overlay). Local gates on the same head: typecheck clean, full suite 2628 pass / 0 fail / 10 skipped, build success.

Nothing else outstanding on my side — remaining pre-merge items are yours: (1) calls on P2/P3/P4 (accept-with-docs or fix), (2) rebase onto current master (9 commits behind; trial merge-tree is conflict-free) then re-run gates. Ping me with the calls and I'll handle the rebase / any fixes.

中文摘要:d8bb1654 的 CI 已全部通过(11/11),本地门禁同样全绿;待你对 P2/P3/P4 拍板并 rebase 最新 master 后即可合并。

@ranxianglei

Copy link
Copy Markdown
Owner Author

Deep review: ✅ merge. Verified on the triple-stacked tree (#1367+#1361+#1360 — all pairwise and triple merges clean; typecheck + full suite 2659 tests / 0 fail on the stack).

Design holds up. The lifecycle queued → attached (snapshot keeps items in the queue) → delivered|dropped(reason) is the right shape: the ack is committed at tool-response time, so every loss path must be observable rather than silent — and all four windows from the issue are covered with dedicated tests (W1 restart reconcile via the ccrReconcilePending one-shot — correctly decoupled from restored, which getSession() clears before prepare; W1b re-retrieved carrier survives; W2 post-drain failure drops + logs, idempotent; W3 disarm terminates observably AND preserves the #1273 no-stale-flush-after-re-arm guarantee; W4 TTL expiry loud with range-restore riders exempt).

Points I probed specifically:

  • Proxy lane drainPendingRetrievals = commit-in-place — correct, that lane composes ack+injection inside ONE response flow, so there is no cross-request window there.
  • prepare catch drops attached refs with reason "prepare failed; forwarded unprocessed" — right: the unprocessed body carries no injection, so dropping + corrective note is honest.
  • Ledger in session.metadata round-trips to disk unchanged; carrier stays in-memory; resetSessionCompression clears both + drops notes (rebase = expected loss, not a restart drop) — the misattribution guard is the right call.
  • commitRetrievals reassigns session.pendingRetrievals; all readers go through carrierOf(session) — no stale reference.

Non-blocking observations (no action needed):

  1. A restarted session whose next request lands on the proxy lane never consumes ccrReconcilePending (reconcile runs only in the plugin-lane prepare) — pre-restart plugin-mode ledger rows sit in metadata until the session touches the plugin lane again. Bookkeeping only, no model-visible effect.
  2. Two concurrent same-session requests can both attach the same refs and both deliver — the model sees the full text twice (token cost, not loss; ref-set filtering makes the second outcome a no-op in the ledger). Rare in practice.

Docs: BILI_CCR_RETRIEVAL_TTL_MS documented in CONFIGURATION.md (default 600000, 0 disables, plugin-lane only). ✅

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Thanks for the deep review and the triple-stack verification (#1367+#1361+#1360) — glad the lifecycle design held up under probing.

On your two non-blocking observations: agreed on both. (1) Proxy-lane-only sessions keeping stale ccrReconcilePending ledger rows is bookkeeping-only by construction — no model-visible effect, no false note can fire from an unconsumed flag. (2) The concurrent double-attach is inherent to snapshot semantics and self-limits exactly as you describe (second outcome is a ledger no-op; cost is duplicated tokens, not loss).

I've also addressed my own open item from the review thread: the PR body now carries a "Known bounded residuals (accepted at review)" section recording all five — plugin-lane 2xx-header settlement point, user-message note channel, proxy commit-at-drain, reconcile scope, and concurrent double-attach — so the merged PR is honest about what it does and doesn't close. No code changes involved.

I can't merge PRs — system rules forbid it. Please merge yourself: #1367 (note mergeable_state: behind — rebase onto current master first if you want a clean merge; trial merge-tree was conflict-free, so it should be trivial).

中文摘要:已按你的深度评审结论收尾——PR 描述补充了「已知有界残留」一节(五项残留全部留痕),无代码改动;合并操作按系统规则需你本人执行,合并前建议先 rebase 最新 master。

@ranxianglei
ranxianglei merged commit 2a0eff7 into master Sep 26, 2026
11 checks passed
@ranxianglei
ranxianglei deleted the 2026-09-26_ccr-retrieve-delivery-lifecycle branch October 15, 2026 05:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(ccr): plugin-mode retrieve acknowledgement can outlive its volatile full-text delivery

1 participant