Repository navigation
[PR] fix: gate failed-compress record cleanup on real folds + attribute its cache breaks (#3769) - #3784
[PR] fix: gate failed-compress record cleanup on real folds + attribute its cache breaks (#3769)#3784ranxianglei wants to merge 20 commits into
Conversation
📦 Built Package ArtifactBranch: Option A — Install this exact build (recommended)npm install -g billion-context@0.0.4615-pr.3784.49de9f3This exact version is immutable and reproducible. The rolling Option B — Download artifact
tar xzf billion-context-pr3784.tgz
npm install -g packageInstall instructions are refreshed on each push. |
|
CI 红了 |
(ai: vllm-qwen/qwen3.8-27b) 查重: 不重复(本 PR 的审核线程;相关 #3003/#3670 已在 PR 正文交叉链接)。 「CI 红了」的直接答案:三层——两层是本 PR 侧问题(已修),一层是仓库级基础设施(与本 PR 无关)。
行为变更披露(PR 正文已有,此处重申):移除 keep-two 轮换 → failed compress pair 保留到一次 fold 提交,之后只清理 fold 起点及之后的一次性、持久化条目;安静轮不再重写 mid-history(这正是 #3769 的修复),最近失败反馈保留,「避免重复空压缩」的回路保护不受影响。 八问:
判定:三个审核修复 commit 已推送(bc4f4cb / 4720ac5 / a9333f9,各自独立标注);core 矩阵重跑结果出来补最终结论。kernel 变更按规则需人工审核,我不合并。 |
|
复核最新提交 目前建议处理以下事项,按优先级排序。不需要重构现有 fold-gated 方案,尽量保持改动最小化。 P0(必须):旧会话升级兼容性旧版 kernel 通过 当前实现似乎没有将旧状态中的
建议:
这直接关系到 #3769 的核心目标,建议在合并前解决。 P1(必须确认):CI 失败原因最新提交
建议检查具体失败断言,确认是否由本 PR 引入。 如果属于已有的不稳定测试或无关故障,说明原因即可;如果是实际回归,则需要修复。 另外,之前显示绿色的运行存在测试矩阵被跳过的情况,不能直接作为通过依据。 P2(可选):完善 orphan-sweep 账本归因当前
这可能导致账本统计不够准确,但不直接影响缓存历史的实际处理逻辑。 如果希望完整满足 #3769 原方案中的账本验收要求,建议:
如果当前目标是尽快修复核心缓存问题,这部分可以拆成后续 PR,不必阻塞本次合并。 P3(可选):无关 CI 配置问题
如果确认这是既有问题、与本 PR 无关,可以单独处理,不建议为此扩大当前 PR 的修改范围。 建议合并条件:
其余项目可以后续跟进。当前核心设计不需要再做大范围调整。 |
(ai: vllm-qwen/qwen3.8-27b) 收到复核意见,P0/P1 正在处理中,先同步进展(结论与完整八问在修复落地后一并给出): P0 已定位到比「缺少迁移」更深一层的根因——旧状态加载本身没问题(
修复(最小化,不动 fold-gated 方案本身):
P1:三个失败 leg 的断言全部是 P2(账本归因位置细化)、P3(ci-gate-plugins secrets)按你的意见留作后续,不阻塞本次合并。提交推送后立即补完整回复。 |
|
冲突 ci红了 |
(ai: vllm-qwen/qwen3.8-27b) 收到,正在处理。进度同步(细节等全部收敛后按八问回复):
全部定位+验证完成后一次性推分支并给完整八问结论。 |
(ai: vllm-qwen/qwen3.8-27b) 状态同步:冲突已解、CI 红的根因已修完并本地验证全绿;分支推送暂时被沙箱出网通道卡住,推上去后我立即确认 CI 并给正式八问结论。 CI 红的是什么(旧 head 上就红,与 master 冲突本身无关):
怎么修的:
本地验证(新 head 定向跑,全量交给 CI):kernel 50/50;host 定向 128/128——含 CI 红的四个文件、CFP 11/11、D-lane 12/12、cache/storage 相邻套件 38/38;typecheck 干净。两个 commit 已打好(fix: kernel carrier 豁免 / test: CFP+upgrade-compat 对齐)。 当前卡点:沙箱到 github.com 的出网代理持续重置连接约 35 分钟(api.github.com 正常、直连不通),push 一直失败。后台每 45s 重试中,通了马上报。 |
(ai: vllm-qwen/qwen3.8-27b) 补充一条:刚才又收到一次冲突通知——master 在这期间前进了 10 个提交(#3733 dsh-authored-compaction、#3837 headroom log 命名、文档类)。已逐文件比对:新增改动只落在 src/knobs.ts、src/server/handle.ts、src/server/prepare-*.ts、src/server/dsh-authored-compaction.ts(新)、tests/dsh-authored-compaction.test.ts(新)及文档——与本 PR 触碰的文件零重叠,rebase 到最新 master 应当无冲突。 当前唯一阻塞仍是出网通道故障(约 90 分钟了,git 推送持续被重置,API 通道正常)。恢复后我会一次性完成 fetch → rebase 到 39fb841 → 定向验证 → force-push,然后按八问收尾。代码侧已全部就绪(本地 tip |
|
继续 |
The kernel kept only the newest TWO unmatched compress call+result pairs and hid older ones on EVERY turn, decoupled from real folds. On a third failed compress this deleted the oldest pair mid-history, rewriting an earlier cached prefix at full cost with no fold behind the break. Replace the keep-two rotation with fold-gated, position-aware, persistent sweeping: orphans stay visible between folds (append-only, cache-friendly); a committed fold stamps pendingOrphanSweepMinRef (earliest new-block startRef) and the next pass hides only orphans at/after that ref, so the sweep rides the fold's own rewrite (cache-free) while earlier orphans (still-cached prefix) survive; swept ids persist in sweptOrphanCallIds (monotonic) so a resumed session never resurrects them. Bump acp-kernel 0.0.136 -> 0.0.137.
…3769) The #3769 seam landed in the 'no fold/switch/restart attribution' bucket because the break came from bili's own cleanup of failed-compress records, not a fold. Add an orphan-sweep attribution channel to the cache ledger: detect growth in the session's sweptOrphanCallIds set per sample and, when a mid-history break has no other claimant, label it prov='orphan-sweep' (count + missed tokens) instead of leaving it unexplained. A real fold claims its own break first, so there is no double-counting. New counters normalize through getCacheLedger for persistence compatibility.
types.ts and two kernel test files sit on a prettier-clean master baseline; the #3769 additions landed over-indented by one space. Normalized to the surrounding 2-space style so the diff stays confined to the feature. No behavior change; typecheck + targeted kernel/host suites re-run green.
…eview fix) Review found a defect in the fold-marker computation: block-boundary specs (tier distillation compresses bN..bM) store startRef="bN" verbatim on the new block, so Number(startRef.replace(/\D/g,"")) parsed a BLOCK number, not the earliest covered MESSAGE ref. In long sessions that marks the sweep far too early: the next hide pass deletes failed pairs sitting in the still-cached prefix before the fold's own divergence point — re-introducing exactly the #3769 mid-history cache break on every tier-distillation fold. Compute foldStartRef as the min refNum over each new block's effectiveMessageIds via state.messageRefs.byRaw instead; ids missing from the map (pruned) contribute nothing, which only under-sweeps (safe direction). refNum is exported from hide-consumed.ts so both consumers share one "unknown" definition. Regression test 6 in issue3769-orphan-sweep.test.ts: m-boundary fold then block-boundary fold asserts pendingOrphanSweepMinRef === first covered ref (not the block number) and that the earlier pair stays visible. Verified to fail against the pre-fix code.
Review found the two new kernel state fields were dropped on every session load: buildRecord persists session.state wholesale, but the load path rebuilds it through mergeState's explicit field whitelist, which predated sweptOrphanCallIds / pendingOrphanSweepMinRef. Losing sweptOrphanCallIds resurrects already-cleaned failed pairs after a restart — a mid-history rewrite (cache break) that violates the PR's own persistence acceptance case. Losing the pending marker only defers a sweep by one fold (conservative). Extend the mergeState whitelist with both fields, degrading to undefined for old persisted states exactly like every other optional field.
…-gated sweeps (#3769) Three stale pins broke under the approved keep-two -> fold-gated behavior change: - stub-resubmission-guard (#2749): the fixture applied folds without a call id, unlike production (src/loop/core.ts passes callId), so the resent call classified as orphan and the new sweep hid it. Pass "call-b1" at each first fold; all summary-integrity assertions unchanged. 5/5 green. - cache-friendly-plugin: the two anthropic geometry subtests pinned the old KEEP_LAST_ORPHANED=2 rotation (oldest pair pruned on later turns). Rewritten to pin the new contract: failed pairs stay visible between folds, every anchor stays stripped while its own pair rides, quiet turns append-stable. A live-server probe plus the passing drop-pair/main-matrix subtests confirm the geometry expected values. Header docs updated to match the new mechanism. 11/11 green. - storage-render goldens: regenerated via scripts/update-storage-render-goldens.ts. Diff is additive only — the persisted envelope gained lastSweptCount (cache ledger) and the two new kernel state keys (empty digests); state/blockContents/stats section digests unchanged.
Pre-#3769 states persisted the keep-two rotation's hidden set as message refs (hiddenOrphanRefs); the new hide pass ignored it, so upgrading would resurrect rotation-hidden failed pairs mid-history and break the cached prefix again. Honor the legacy set on upgrade; rememberHiddenRef re-records those refs every pass, so the clause self-perpetuates without any state migration.
syncBlocks deep-clones CompressionState with an explicit field list that missed hiddenOrphanRefs / sweptOrphanCallIds / pendingOrphanSweepMinRef. It runs BEFORE the hide-compress-calls node in every processTurn, so all three ledgers were wiped each turn: the fold-gated sweep never fired end-to-end and the legacy upgrade-compat clause was unreachable. Carry them exactly like cloneState does (lastPassIds stays excluded: its only reader runs pre-sync and the field is re-stamped at processTurn exit).
…P0) Two processTurn-level pins: an upgraded pre-fold-gate state keeps its rotation-hidden orphan hidden across quiet turns (legacy hiddenOrphanRefs clause), and a fold marker set by applyCompression survives the full pipeline to the hide node (sync-blocks carry) so the fold-gated sweep actually fires end-to-end.
applyForkAdoption rebuilds session state from the parent on resume/fork replay; without carrying hiddenOrphanRefs and sweptOrphanCallIds, failed compress pairs already hidden pre-upgrade or already swept would resurface on replay, rewriting mid-history at full cache cost. Both ledgers are monotonic sets re-recorded every pass, so the carry self-perpetuates.
Real-server e2e: drive a session with a failed compress pair, persist it, patch the record back to an old-shape state (hiddenOrphanRefs set, no #3769 fields), reload under a fresh proxy, and assert the previously-hidden pair stays out of the outbound history while new turns append stably.
With the sync-blocks carry in place the sweep fires end-to-end: a failed compress pair appended at a fold step is swept by that step's own model request, so it never rides inbound and its anchor carries the summary for good (zero-carrier fail-safe, #1567). Re-pin the main matrix (all four wires), the swallow/back-to-back geometries (wide fold absorbs the alpha carrier; FOLD_E tail hits the protected zone), the drop-pair subtest (client-side drop is now a wire no-op) and the 3-concurrent-folds pin (carriers stack one per fold, stable thereafter), from observed live-server bodies. Google pair-absence proxy switches from the topic (which the carrier embeds) to the fold-spec refs that only the re-sent functionCall args carry.
…3769/#1567) The fold-gated sweep hid EVERY orphan-fate compress pair at/after the fold's start ref — including successful echoes whose call id never matched the server-minted block id. A re-sent pair quoting an ACTIVE block's range is the plugin lane's summary carrier (KDD#7), so hiding it strands the summary: the #1999 double-carrier test, the plugin session-lock e2e, #1426 mode-upgrade proof and #1638 notification pin all went red. resolveFate's 'orphan' verdict is id-based and cannot see success; the sweep branch now additionally requires the call to quote NO active block range (quotesActiveBlock: both arg shapes — content[] entries and flat top-level startId/endId — mirroring the host stripKernelSummaries carrier recognition). Genuinely failed records and dead-block pairs (swallowed by a wider fold) still sweep at the fold boundary, per the approved #3769 design.
…carrier regime and #3818 full-mode default (#3769 follow-through) cache-friendly-plugin: the previous rewrite pinned the pre-floor-20 sweep regime (successful echo pairs hidden at their fold step, anchors carrying). That contradicts KDD#7 (plugin lane's wire carrier IS the tool call; the agent view never renders the kernel acp_summary), the approved #3769 scope (failed-record cleanup only), and the suite's own P6 dsh-rebase subtest, which already pinned pair-rides/anchor-stripped. Re-pin to the verified carrier regime across all four wires: pairs ride from their fold turn on (summary twice per riding pair: args + receipt), SUM_MARKER zero while any pair rides, drop-pair surfaces exactly its anchor once then stays append-stable (#1567 fail-safe), dead-block pairs still sweep at the swallow boundary (#3769), google lane expectations follow the #3159 canon-shift. issue3769-upgrade-compat: since #3818 flipped the storage default to full, the unified index row is the load path — patching only the JSON envelope left a stale row the loader served instead, so the simulated old-kernel record now goes through the store (loadSync + flushSync dual-writes both copies). Teardown now closes both proxies even when an assert throws (previously a mid-test failure hung the runner).
0a37b5b to
48637ed
Compare
…rts gate) #3811 exported both the per-URL primitive and the text-level sink, but only maskUrlsForPublish is imported outside log-mask.ts (src/web/diagnostics.ts). The singular helper is internal plumbing of the plural one — un-export it per the gate's INTERNAL_ONLY remedy. No behavior change.
The committed CONFIGURATION.md/zh-CN carried two stale duplicate rows around storage.unified/storage.sqliteAutoDownload (pre-#3818 wording plus the current rows). tools/gen-config-docs.mjs generate removes exactly those two rows per file; no other artifact changes.
88e18ae to
44ce707
Compare
The gate also re-derives the kernel's OWN lockfile: it was still at 0.0.136 (stale since the original 0.0.137 bump). Two version lines only.
(ai: vllm-qwen/qwen3.8-27b) PR #3784 CI 修复结论:三条红灯已在分支上全部转绿(已本地验证) 先给结果: 排查与分层:
关键判断:我把后两条门禁在基线 master 进展变化: 我推 kernel lock 修复时远端分支已前进,另一个 agent 并行推了 4 个提交把三条全修好( 结构化八问:
判定:CI 已在分支上转绿(待远程重跑确认),无需我再动手。 |
|
CI 红了 原因分析问题出现在测试的第二阶段: 这里 检查
而正常的 因此更可能是测试没有按照正常恢复路径加载记录,而不是 fold-gated 清理逻辑本身出错。 不过仍需要修改并重跑才能最终确认。 建议修改在测试中使用持久化记录里的真实 ID 和路由元数据: 同时给 也可以直接通过正常启动恢复流程测试,但应保持 最终判断当前 CI 红灯确定来自 PR 新增的测试,但还不能据此证明生产代码存在回归。 当前测试在旧会话状态成功恢复之前就失败了,因此根本没有运行到验证旧 orphan 是否重新出现的阶段。 建议先修正加载方式,再重跑。如果后面的 相关代码:失败测试第 179 行 · 持久化加载逻辑 目前不用大改 PR,先针对这一处测试恢复前置条件修复即可。 |
…id (#3769) Phase 2 built a fresh un-booted SessionStore and called loadSync(LABEL) with only the id. That store has neither a populated unified index nor a walked diskFiles set, and Phase 1 wrote the record under a protocol-namespaced path (<protocol>/<host>_<sha>/), so the id-only probe falls back to _unknown/ and returns null. The test died at 'record hydrates before the upgrade turn' before ever reaching the hidden-orphan assertions. Under the #3818 full-mode default this hid behind the unified index whenever it happened to be populated, so it flaked across CI shards instead of failing deterministically. Pass the record's persisted {protocol, upstreamOrigin} to loadSync (mirroring getSession's recovery path): loadNamespacedPool then resolves the exact file by direct path regardless of index/diskFiles state. Extend PersistRecord.meta with those fields. Verified green in legacy-only AND full modes; the hidden-orphan assertions (args/result stay hidden, tail rides) pass, so no production regression - the red light was the test precondition alone.
(ai: vllm-qwen/qwen3.8-27b) 结论:CI 红灯已修并推上分支(新 tip 背景:这条 head 上静态门禁(lock-sync / unused-exports / docs-sync)已全部转绿,只剩 3 个 test 分片红( 先回应 @kuviki 的分析:方向完全正确。我独立复核后确认你的判断: 根因(分层):
修法: 给 关键验证(不止本地跑绿):
结构化八问:
判定:修复已推上 PR 分支 tip |
(ai: vllm-qwen/qwen3.8-27b) 这条分析我在上一条回复里已经处理完并推上分支了——你建议的改法正是我落地的方案,不用再重复提交。当前状态:
结论:无需再动 PR。等 test 分片重跑转绿即可合入;若仍有分片红我再定向排查。 |
…eline the #3769 proof to the fold-gated contract (#3939) - truncate-tools: build truncatedIds in the loop instead of spreading Map.keys() — the iterator spread degrades the symbolic translation of the 3691 harness (verif refuses to certify on degradation). - hide-consumed: hoist state.hiddenOrphanRefs ?? [] — .includes on a possibly-undefined receiver degrades the same way. - verif/symbolic: fold Number(constString) to its numeric constant; the old {kind:any} made Number.isFinite collapse to constant false, silently deleting the refNum-gated fold-sweep path from every model (real code fires it; the model could not — engine gap, not kernel bug). - verif-bug3769 harness re-baselined onto the #3784 fold gate: Q1 (append-only growth hides r1) is now UNSAT — the fix holds; Q1b proves the sweep is still derivable THROUGH the fold marker (judge not vacuous); replay pins hidden=0 without a pending fold and newlySwept=[c1,c2,c3] with one. - kernel 0.0.142 -> 0.0.143 (both locks synced). verif: 15/15 green with cvc5 (subset-guard, symbolic-smoke, 3691, 3769, 3805, 3676, generic, induction). kernel suite 1357/1357.
…ch (ranxianglei#3939 ⑤) absorb-hide, crush and filter keepLastOnly re-derive their output every pass from state/context (absorb records growing, crush responding to token pressure, keep-last-only demotions as new occurrences arrive) — each change flipped OLD message bytes on an otherwise quiet turn (the ranxianglei#3805 prefix-cache breaker class). One structural latch now covers all of them: post-transform bytes are latched per padded ref at first application and replay verbatim; refreshes ride fold boundaries (applyCompression resets the map), the absorb action itself (instant- reclaim contract kept), and config edits (digest release — enable/ disable applies on the edit turn, never silently later). ccr needs no latch (replace-once-at-arrival, byte-stable by design ranxianglei#1460). Placed after hide-compress-calls (the ranxianglei#3784 sweep keeps its own gate) and before recommend/nudge/truncate/render so downstream judges see the view the model actually sees. Carried across sync/persist/fork/plugin/ ingest; storage-render goldens pin the two new state keys (undefined valued), wire bodies unchanged.
kernel-change: replace keep-two failed-compress rotation with fold-gated sweeping (#3769)
Why the host side cannot solve it: the premature-cleanup timing lives entirely inside the kernel.
kernel/src/hide-consumed.tsdecides which unmatched compress call/result pairs to hide, on everyprocessTurn, decoupled from whether a compression actually committed. The host only recognizes compress tool names (isCompressTool); it cannot defer or re-time the kernel's own hide pass. So the timing fix has to land inkernel/.Closes #3769. Related (same family, independent mechanism, cross-linked in-thread): #3003 / PR #3670.
What this does
Kernel (the fix). The kernel kept only the newest two unmatched (failed) compress call+result pairs and hid anything older on every turn, regardless of whether a fold committed. On a third failed compress it deleted the oldest pair from mid-history — rewriting an earlier cached prefix at full cost, with no fold behind the break. That is exactly the seam #3769 reproduced (first divergence byte 89501,
bodyClass=mid, ~182k unattributed tokens).Replaced the keep-two rotation with fold-gated, position-aware, persistent sweeping:
applyCompressionstampspendingOrphanSweepMinRef= the earliest start-ref of the newly created blocks. On the next pass the kernel hides only orphans at/after that ref, so the small deletion rides the fold's own rewrite (cache-free) while earlier orphans (still-cached prefix) survive.sweptOrphanCallIds(monotonic), so a resumed session never resurrects them.Host (observability companion). The #3769 seam landed in the cache-ledger's "no fold/switch/restart attribution" bucket because the break came from bili's own cleanup, not a fold. Added an
orphan-sweepattribution channel: detect growth insweptOrphanCallIdsper sample; when a mid-history break has no other claimant, label itprov="orphan-sweep"(count + missed tokens) instead of leaving it unexplained. A real fold claims its own break first → no double-counting.Behavior change (old → new) — MUST READ
KEEP_LAST_ORPHANED = 2— the two newest failed pairs kept, all older ones hidden every turn.Config surface
None. No new env var / config field / CLI flag (per the issue's explicit request). Two new internal kernel state fields (
sweptOrphanCallIds,pendingOrphanSweepMinRef) ride existing session persistence — additive; old states degrade to "nothing swept".Verification
issue3769-orphan-sweep.test.tscovers all four acceptance cases (quiet-turn append-only ⇒ no mid-history rewrite; fold sweeps at/after start, keeps earlier; persistent across turns/restart; no per-failure rolling cleanup). Updated the three tests that pinned the old keep-two contract.issue3769-orphan-sweep-ledger.test.ts(positive attribution, no-overfire gating, resume-baseline).Review notes
src/cache-ledger.tsis a hot file; the additions are confined to the seam-detection/report regions (disjoint from master's recent nudge work).Addendum — conflict resolution (rebase onto current master) + carrier exemption (#3769/#1567)
Rebased onto master (conflict with #3803's effort-switch ledger fields resolved as union in
src/cache-ledger.ts; storage-render goldens regenerated). Three follow-up fixes landed on top:kernel/src/hide-consumed.ts, kernel 0.0.137→0.0.138). The original fold-gated sweep hid every orphan-fate pair at/after the fold start ref — including successful echoes whose client-side call id never matches the server-minted block id. A re-sent pair quoting an ACTIVE block's range is the plugin lane's summary carrier (KDD#7); hiding it strands the summary and broke the # [Bug] DSH 插件模式 + OpenAI Responses 上游:compress 后摘要载体以developer角色插入历史中段,严格单-system 后端 400「System message must be at the beginning」 #1999 double-carrier pin, the plugin session-lock e2e, feat(web): full UI rewrite — read-only overview / sessions / config / connect (#1420) #1426 mode-upgrade proof and [Bug] bili 启动 OMP 后缓存命中率 82.7%,连续请求多次骤降至 4992 cached tokens #1638 notification pin. The sweep branch now additionally requires the call to quote NO active block range (both arg shapes —content[]entries and flat top-levelstartId/endId— mirroring hoststripKernelSummariescarrier recognition). Genuinely failed records and dead-block pairs (swallowed by a wider fold) still sweep at the fold boundary — the approved design is unchanged. New kernel test pins both directions (exemption + negative control).tests/cache-friendly-plugin.test.ts). The previous rewrite pinned the pre-tightening sweep regime (pairs hidden at their fold step, anchors carrying), contradicting KDD#7, the approved scope, and the suite's own P6 dsh-rebase subtest which already pinned pair-rides/anchor-stripped. Now pinned across all four wires: pairs ride from their fold turn on (summary ×2 per riding pair: args + receipt), SUM_MARKER zero while any pair rides, client drop surfaces exactly its anchor once then append-stable (plugin mode: in-place acp_summary never stripped when the client's own compress pair rides inbound — double summary carrier on every post-fold turn #1567 fail-safe), dead-block pairs still sweep at the swallow boundary ([Bug] MCP failed compression record rotation prematurely rewrites the history prefix, triggering expensive cache recompu | [Bug] MCP 失败压缩记录轮换提前改写历史前缀,触发大额缓存重算 #3769), google lane expectations follow the Vertex rejects tool responses when Bili appends trailing nudge text #3159 canon-shift.tests/issue3769-upgrade-compat.test.ts). Since Storage wrap-up: 6 fixes + switch default mode to full (single PR) | 存储收尾六修 + 缺省模式翻 full(一个 PR 统一处理) #3818 flipped the storage default to full mode, the unified index row IS the load path — patching only the JSON envelope left a stale row the loader served instead. The simulated old-kernel record now goes through the store (loadSync+flushSync, dual-writes both copies). Teardown also closes both proxies even when an assert throws (previously hung the runner).Local targeted verification on this head: kernel 50/50, host targeted 128/128 (incl. all four formerly-red files, CFP 11/11, D-lane snapshot-render 12/12, adjacent cache/storage suites 38/38), typecheck clean. Full-suite verdict = CI on this head.