Skip to content

fix: detect silent fold loss when resent history ids drift from compressed blocks (#1195) - #1247

Merged
ranxianglei merged 2 commits into
masterfrom
wt-1195-ework-daemon
Sep 26, 2026
Merged

ranxianglei merged 2 commits into
masterfrom
wt-1195-ework-daemon

Conversation

@ranxianglei

@ranxianglei ranxianglei commented Sep 24, 2026 •

Copy link
Copy Markdown
Owner

#1417 批次审查注记(2026-09-27):issue #1195 已由 #1328(响应侧 overflow rescue+refold)修复并关闭。本 PR 的定位随之收窄为 #1328 的观测补充:输入侧 fold-drift 归因(为何此会话反复被 rescue/400 发生前就发现折叠失效),与 #1328 互补而非替代。head 已 rebase 至当前 master 并修正告警措辞(覆盖 mutation 与良性客户端删除两种场景)。

Context

From #1195: after a successful pi compression (2 blocks, ~51K tokens saved), the very next request fails with Prompt is too long.

While tracing that report I found a silent failure path worth instrumenting regardless of which mechanism fired in that specific incident (see #1148/#1102 for the fork-class cousins already fixed in v0.1.148):

Message ids are content hashes (deriveMessageId). When a client resends history whose bytes changed (resume/re-serialization, edit, duplicate-cluster shift), compressed-range ids no longer match the block's covered set, and those messages silently re-enter the wire unfolded — while the compress result already booked their savings into lastInputTokens, blinding preflight/nudge decisions for subsequent turns. Today this path produces zero log output.

What this PR does (log-only diagnostics)

  • src/session.ts: new foldCoverage(coveredBefore, liveRawIds) — returns {expected, matched} when some covered ids are missing from the resent history, null on full coverage.
  • src/server.ts: all four main prepares (anthropic/openai/google/responses) now snapshot the active blocks' covered ids before processTurn when the previous turn ended with a pending fold (stats.pendingFoldUsage === true), then compare against the resent message ids after state adoption. A mismatch logs one line:
    [acp-drift] fold coverage mismatch: matched/expected covered message id(s) present in resent history — N message(s) from the just-compressed range did not fold out of the wire ...
  • Guards so side requests/title generation/compaction triggers never trip it (msgs.length >= REWRITE_MIN_INCOMING_TOTAL, !isTitleGen, !isCompactionTrigger).
  • tests/fold-drift.test.ts: 4 regression tests (pure helper semantics + a server-mirroring materialization check: stable resend silent, drifted resend fires).

Behavior change disclosure

None. No wire change, no config change, no persistence change, no threshold/default/timing change. The only observable difference is a new warn log line in sessions where a completed compress fold demonstrably did not materialize. Both compression modes unaffected (the check is mode-agnostic; plugin-mode warnings already flow through the same logger).

Verification (rebased head c1ea178 on master 570a2ea / v0.1.148)

  • npm run typecheck ✅
  • npm run build ✅
  • node --import tsx --test tests/fold-drift.test.ts → 4/4 ✅
  • Full npm test → 2547/2557 pass, 0 fail, 10 skipped ✅ (green on the rebased head)

Scope note

This is the observability half of #1195. It makes the drift mechanism self-diagnosing in production logs. If the reporter's evidence (upstream error body token counts + bili.log around the failure) confirms this mechanism as the actual cause, the targeted mitigation will land on this branch before merge; if a different mechanism is confirmed, this PR still ships as a pure win (it closes a real silent blind spot).

@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: wt-1195-ework-daemon (6bb6056)

Option A — Install from npm PR tag (recommended)

npm install -g billion-context@pr-1247

Each push to this PR publishes a new version under the pr-1247 npm tag.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr1247.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Review of #1247 (log-only fold-drift diagnostics for #1195) — reviewed code against both the PR head (0d1d496e) and current master (d17d81fb / v0.1.146), ran the full pre-flight myself. Verdict: mergeable.

① Original issue / direction. #1195's incident analysis identified id-drift silent fold loss as the leading suspect mechanism: content-hash ids drift when a client resends mutated bytes, covered ids stop matching active blocks, those messages silently re-enter the wire unfolded while their savings were already booked into lastInputTokens — blinding preflight/nudge with zero logging. This PR instruments exactly that decision point (snapshot covered ids before processTurn, compare against resent ids after adoption) instead of guessing at the mechanism. Correct root-cause-first sequencing: make the path self-diagnosing now, land the targeted mitigation once the reporter's evidence (launch mode, window, upstream error token counts, bili.log around the failure) pins the cause. Direction approved.

② Regression inventory. Touched paths: the four main prepares in src/server.ts + one export/helper in src/session.ts. Checked each existing behavior they carry:

  • REWRITE_MIN_INCOMING_TOTAL: value unchanged (10), only const → export const; existing consumers unaffected.
  • All four prepare insertions are strictly read-only (a Set built from state.blocks) plus one conditional warn — no mutation of parsed/state/session, no wire shape, config, persistence, threshold, default, or timing change. The behavior-change disclosure ("None") is accurate.
  • Guard correctness verified in code: side requests (title-gen etc.) bypass the prepares entirely via the #388 sidePassthrough gate, so the !isTitleGen guards are defense-in-depth; prepareAnthropic lacking an isTitleGen var is therefore safe (length guard suffices). !isCompactionTrigger correctly scoped to prepareResponses where that concept exists.
  • Mode-agreement holds: plugin mode resends the agent's own compress call/result each turn, so covered ids stay present on healthy flows — no new warnings expected there; proxy mode is the primary beneficiary.
  • Interaction with orphan-gc.ts (reapOrphanBlocks, post-turn, full-orphan streak ≥ 3): complementary, no conflict — this check is pre-turn and catches partial gaps orphan-GC deliberately ignores.
  • One non-blocking observation: after a codex native compaction truncates the client's own history, a still-active block whose covered ids were compacted away could fire one warn line whose wording ("did not fold out of the wire") is slightly off (the messages were deleted, not re-entered). Cost is a single potentially-misworded log line in an unusual flow; boundary reconciliation normally prunes first. Not worth changing before merge.

③ Code. foldCoverage(coveredBefore, liveRawIds) is pure and minimal; null on empty/full coverage vs {expected, matched} otherwise is the right contract. Snapshot placement before processTurn is correct for the stated reason (syncBlocks may deactivate fully-drifted blocks inside processTurn); the check then uses the captured value, so the later pendingFoldUsage clear can't mask it. Log line carries matched/expected + missing count — actionable, no secrets, routed through the standard log callback (logger rule respected). No as any/@ts-ignore, no hex-escape violations.

④ Tests. tests/fold-drift.test.ts — 4 tests, all with teeth: three pin the helper's exact semantics (null on empty/full coverage, exact {expected, matched} counts on partial gap); the fourth mirrors the server capture/check pair through a real kernel processTurn (24-message fixture, m00001–m00007 folded into one live block) and asserts the two outcomes that define this feature: stable resend → silent, drifted resend → fires with matched = expected - 1. It is a genuine regression test for the detector itself, robust to explicit-id vs content-hash derivation. No tautologies.

⑤ Diff. 3 files, +165/−2, single commit, every line on-purpose; no whitespace churn, no lockfile/generated noise. Clean.

Pre-flight (run by me, not just CI):

typecheck build fold-drift full suite
PR head 0d1d496e (v0.1.145 base) ✅ ✅ 4/4 2418/2423 pass, 2 skip, 3 fail
rebased onto master d17d81fb (clean cherry-pick, no conflicts) ✅ ✅ 4/4 2437/2442 pass, 2 skip, same 3 fail

The 3 failures are the tests/claude-native.test.ts linux live-E2E trio (ps-walk race kill ESRCH + two 60s proxy-up timeouts). I reproduced them standalone on plain master without this PR — identical set — confirming they're the shared-sandbox flake already tracked separately (#1248), zero overlap with this diff. GitHub CI on the head is fully green (8/8 checks incl. ubuntu 22/24 full suites), i.e. the dedicated-runner arbiter agrees.

Duplicate screening: no other open/closed issue or PR covers drift/fold-coverage detection.

Two housekeeping notes, neither blocking:

  1. Rebase onto current master before merge — base is v0.1.145-era; master moved through v0.1.146 including heavy src/server.ts churn (feat: image pre-compression for screenshot-heavy tool results (#1095) #1211 image compression). I verified the rebased head is conflict-free and green, so a plain rebase should just work.
  2. Body says "From 使用pi 压缩后就提示prompt is too long #1195" rather than Fixes #1195 — defensible here since this is explicitly the observability half and the mitigation is deferred pending evidence, so closing 使用pi 压缩后就提示prompt is too long #1195 on merge would be wrong. Just flagging so it's a deliberate choice.

中文摘要: 这是 #1195 的可观测性半边——在折叠覆盖检查处加了纯日志的 id 漂移检测(不动任何线协议/配置/持久化/阈值);我逐条核对了四个 prepare 路径的既有行为无回归、测试有牙且能钉住本功能、diff 干净,并亲自验证了 PR 头与 rebase 到当前 master 后的双份预检全绿(claude-native 3 个失败在裸 master 上同样复现,属 #1248 已立案的沙盒 flake),CI 8/8 通过——建议作者 rebase 到最新 master 后即可合并。

ework-agent and others added 2 commits September 26, 2026 19:12
…essed blocks (#1195)

Message ids are content hashes (deriveMessageId). When a client resends history whose bytes changed (resume/re-serialization, edit, duplicate-cluster shift), compressed-range ids no longer match the block's covered set and those messages silently re-enter the wire unfolded - while the compress result already booked their savings into lastInputTokens, blinding preflight/nudge decisions for subsequent turns.

Add foldCoverage() in src/session.ts and a materialization check in all four main prepares (anthropic/openai/google/responses): when the previous turn ended with a pending fold (stats.pendingFoldUsage), snapshot the active blocks' covered ids before processTurn, then compare against the resent message ids after state adoption. A mismatch logs one warn line: [acp-drift] fold coverage mismatch: matched/expected ...

Log-only: no behavior change, no wire change. Guards: side requests skipped (msgs < REWRITE_MIN_INCOMING_TOTAL), title generation (!isTitleGen), compaction triggers (!isCompactionTrigger).
…on; reframe as observability complement to #1328 (#1247 review, #1417)
@ranxianglei

Copy link
Copy Markdown
Owner Author

#1417 批次深审 + rebase/修复合并交付

判定:✅ 可合(修后,已修)

干什么/动机:输入侧 fold-drift 归因——foldCoverage() 在每次 compress 后的下一请求快照活跃块覆盖的 message id,processTurn 后比对重发历史;覆盖缺口 → [acp-drift] warn。回答的是:为什么已压缩的消息又回到了 wire 上(内容被改→内容哈希 id 失效→折叠静默丢失)。

定位澄清(重要):#1195 已被 #1328(响应侧 rescue+refold)修复关闭;本 PR 是 #1328 的观测补充(400 发生前归因/反复 rescue 的根因可见),PR body 已加注记重构叙事。与 #1351(传输层失败分类)不同层零重叠。

四轴:纯观测——零 wire 改动、零注入、零节奏影响、无配置。压缩行为/cache/推荐列表/质量全部零触碰。

rebase + 修复(新 head 6bb6056,3 文件 +165/−2)

  1. 5 处冲突按 union 解(master 的 ccrLoopConfig/absorbBlock 新签名 + PR 的 snapshot 块)
  2. 措辞修正:原文 "did not fold out of the wire" 把客户端删除/裁剪(良性)也暗示成折叠丢失——改为明示两种场景(mutation=真丢失 / deletion=良性)
  3. 4 个 prepare*(Anthropic/OpenAI/Google/Responses)落位一致,guard 与各格式能力相符

门禁与变异

typecheck ✅ / 定向 4/4 ✅ / 2812 测试 0 败 / build ✅(reviewer 在 rebase head 实测)。变异:检测器哑火→2 断言红;强制假阳性→3 断言红——测试有牙。

非阻塞

#1328 rescue 路径重跑 prepare* 时同一 drift 事件可能双 warn(纯日志噪音)。

CI 绿后可合(纯观测,不在 auto-merge 禁区)。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Round-2 verification at rebased head 6bb6056e — confirmed, ready for human merge.

I did not take the pre-flight claims on faith; ran them myself at the new head (worktree detached at 6bb6056e, base = current master tip c030b566, so the branch is no longer stale):

Check Result
npm run typecheck ✅ clean
npm run build ✅ success
tests/fold-drift.test.ts ✅ 4/4
Full npm test ✅ 2812 tests, 2800 pass, 0 fail, 12 skip — matches your claim exactly (claude-native flake absent this run)
GitHub CI @ 6bb6056e ✅ 13/13 checks success (ubuntu/windows 22+24 full suites, codex/dsh/pi/opencode e2e lanes, hermetic registry e2e, version-guard, build-artifact)

Wording fix verified in code (all four prepares, src/server.ts:2770/2975/3212/3458): the warn now reads …covered id(s) missing from resent history — mutation (content edit invalidates content-hash refs, fold silently lost) or client-side deletion/truncation (benign, message no longer on the wire); observability complement to the #1328 overflow rescue (#1195) — both scenarios my round-1 note called out are now explicit, and the #1328 reframing lands in the line itself. Good.

Union conflict resolution verified: snapshot still immediately before core.processTurn (now consuming master's loopConfig with the ccr/absorb plumbing), check still after state adoption, guards per format unchanged (!isTitleGen openai/google, !isCompactionTrigger responses, length-only anthropic — safe per the #388 side-request bypass), and master's storeEffectiveAbsorb/storeEffectiveRules/applyAbsorbView sequence intact after each block. Diff remains exactly 3 files +165/−2 — the rebase added no scope creep.

Your non-blocking note stands as-is: the #1328-rescue double-warn is pure log noise on an already-diagnostic path; not worth a pre-merge change.

Regression posture unchanged from round 1: pure observation — zero wire/config/persistence/threshold/timing impact, both compression modes unaffected. Nothing new introduced by the rebase or the wording commit.

Verdict: 可合. CI green, local pre-flight green, diff clean, my round-1 observation fixed. Merge is human-only per system rules — whenever you're ready: #1247

中文摘要: 我在 rebase 后的新 head 6bb6056e(基于最新 master c030b566)亲自复跑了全套预检——typecheck/build/fold-drift 4 项定向/全量 2812 测试 0 失败,与作者声明完全一致;核对了四处 [acp-drift] 日志措辞已区分「mutation=真丢失」与「客户端删除=良性」并带上 #1328 定位、union 冲突解法保留了 master 的 absorb/rules 管线且 diff 仍为 3 文件 +165/−2 无扩散,CI 13/13 全绿——轮次一提出的观察点已修复,纯观测零回归,可以合并。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant