Skip to content

compress: A single nudge may trigger a chain of model-initiated minor folds, and each fold causes the tail of the prom | compress:一次 nudge 可能引发一连串模型主动发起的小折叠,而每次折叠都会让提示缓存的尾部重新计费 #7333

Open

Description

@ranxianglei

Verdict: file this as one bug. Both gaps are on the same code path (applyRanges, the handler for the model's compress calls) and cause the same cost, so it is one issue with a two-part fix. Two code facts from the brief were wrong in 0.1.216 and are fixed in the reports. First, maxFoldsPerRound is in bili's own [preflight] loop, so it is quoted as an example of the guard that is missing here, not as the cause. Second, compress.minCompressRangeChars (about 5,000 characters) exists, so gap (b) is a floor set too low, not a missing floor.
Patch order: first, a minimum-benefit floor inside the existing gate:no-net-shrink check: a small, local change that reuses an existing refusal receipt. Second, a per-nudge fold budget reset by each nudge injection, which needs a maintainer decision because of #5943 and the open PRs #6760 and #6771.
Verification: I checked the code facts in the published 0.1.216 tarball (build 6a641f378) and read #6720, #3444, #6760 and #6771 on GitHub. I did not run the repro steps. The session figures are the reporter's and were not re-measured. The same text is uploaded as bili-fold-burst-bug-report.md.

Title

compress: one nudge can trigger a burst of small model-initiated folds, and each fold re-bills the prompt-cache tail

Environment

  • billion-context 0.1.216 (npm billion-context, CLI bili; dist/build-info.json commit 6a641f378).
  • Claude Code through bili's /bili/ rewrite lane to an OpenRouter-hosted DeepSeek model (deepseek-v4.1-flash). bili resolved the context window at about 968K tokens.
  • Compression tiers T1/T2/T3 enabled, compress.maxContextLimit = 70%, other compress.* keys at their defaults.
  • All 39 folds counted below are model compress calls. bili classifies them as 38 nudge-follow and 1 active.

Summary

In one 1,632-request production session, 10 nudge rounds produced 39 model-initiated folds. That is 3.8 per nudge on average and up to 5 within a single minute. Two of those folds reclaimed only about 2.8K and 3.2K tokens, while the nudge that prompted them recommended about 120K.

Every accepted fold is committed immediately. When folds land in successive upstream requests, each request rewrites the history at a new position, and the provider's prefix cache misses from that position to the end. bili's own cache ledger attributes 619K uncached tokens (about 15.9K per fold) to post-fold rebuild.

bili already protects its own folds against this. The preflight loop commits one serial fold per round outside emergencies, auto-fold has a 4096-token benefit gate (#3444), and bili's nudge text tells the model not to fold piecemeal. The model-initiated compress path enforces none of this:

  • It accepts any number of folds per nudge.
  • It accepts any fold whose net shrink is at least 1 token. The only other floor is the ~5,000-character minCompressRangeChars.

Turning on requireNudge and requireRecommendation does not change either behaviour.

Steps to reproduce

Any tool-capable model works. What matters is what bili accepts, not what the model chooses, so step 4 forces the split.

  1. Start a scratch instance with an isolated config:

    export XDG_CONFIG_HOME="$(mktemp -d)"
    mkdir -p "$XDG_CONFIG_HOME/billion-context"
    cat > "$XDG_CONFIG_HOME/billion-context/billion-context.json" <<'EOF'
    {
      "compress": {
        "modelContextLimit": 200000,
        "nudgeGrowthTokens": 20000,
        "requireNudge": true,
        "requireRecommendation": true
      }
    }
    EOF
    npx -y -p billion-context@0.1.216 bili start --host 127.0.0.1 --port 18787

    requireNudge and requireRecommendation are on deliberately, to show that the strictest existing gates still accept the burst.

  2. In a second shell, run Claude Code against the lane:

    ANTHROPIC_BASE_URL="http://127.0.0.1:18787/bili/<anthropic-compatible-upstream-base-url>" claude

    If bili's Claude plugin is installed on the machine, its SessionStart hook re-pins the base URL at every session start. Use an environment or user profile without the plugin.

  3. Grow the context past the nudge step. Ask the agent to read three or four large text files in full (for example this repo's docs/configuration.md and README.md) until the model receives a compression nudge. acp_status then lists the compressible ranges.

  4. Send: "Fold the largest compressible range that acp_status lists as four separate compress calls: split it into four contiguous sub-ranges, one sub-range per call, and wait for each receipt before issuing the next call."

  5. Watch the proxy log and the dashboard at http://127.0.0.1:18787/__bili/ (Cache economics, and compression recovery per nudge round).

Control: repeat steps 1–3 in a fresh session, then ask for the same span as one compress call carrying all four sub-ranges.

Expected vs Actual

Case Expected Actual (0.1.216)
Four piecemeal calls after one nudge The first call folds. Later calls in the same nudge window are refused (or deferred) with a receipt telling the model to batch, unless the context is in the pressure band. All four are accepted: four [acp-proxy: compress requested 1 range(s): …] log lines, four [Compressed … → 1 block(s), ~N tokens saved.] receipts, no REFUSED.
Upstream prompt cache One post-fold rebuild for the nudge, as in the control. Four rebuilds. Every request after a fold reports fewer cached input tokens, and Post-fold rebuild grows once per fold.
A fold reclaiming ~3K tokens below the pressure band Refused with an error receipt; state not committed, cache untouched (the same shape as today's gate:no-net-shrink refusal). Accepted. Only netDelta <= 0 and the ~5,000-character minCompressRangeChars floor are checked.
requireNudge + requireRecommendation on Not applicable: these gates control timing and scope, not count or size. No effect. The nudge credit is a 20-round window that successful folds do not consume, and splitting a recommended span into smaller pieces is explicitly allowed.

Evidence

One 1,632-request production session. All figures come from bili's own per-session report (Cache economics, and compression recovery per nudge round); derived rows are marked.

Measure Value
Input tokens 319.5M, of which 313.8M cached (98.2%)
Nudge rounds 10
Folds 39 (38 nudge-follow, 1 active)
Folds per nudge (derived) 3.8 average (38 / 10), 5 maximum
Recovery (Σ folded ÷ Σ recommended) 110.5%
Worst burst 5 folds within one minute after a single nudge; two reclaimed ≈2.8K and ≈3.2K tokens, while that nudge recommended ≈120K
Context usage when folding 317,398 / 968,000 = 32.8% (the pressure band starts at 70%)
Uncached input tokens 5.7M = new content 1.9M + post-fold rebuild 619K + unattributed repay 3.2M
Post-fold rebuild per fold (derived) 619K / 39 ≈ 15.9K tokens

Root cause

Code references are to dist/index.js in the 0.1.216 tarball; search for the quoted identifiers.

  1. Model-initiated folds are committed after a zero-benefit check only. In applyRanges (the compress handler), after parsing and the optional requireNudge / requireRecommendation gates, the only benefit test is (reformatted from the bundle):

    if (r.blocksCreated > 0 && touchedBlocks > 0 && netDelta <= 0) {
      // … "Compression refused: no net shrink …"
      return compressResult(receipt, "refused", 0, "gate:no-net-shrink");
    }
    ctx.session.state = res.state;   // commit

    A fold that shrinks the context by one token is committed. The only size floor is the kernel's compress.minCompressRangeChars (old name minCompressRange; default about 5,000 characters, roughly 1.2K tokens of English). It is counted across the whole call and does not apply to ranges that refold existing blocks. A 2.8K-token fold clears it more than twice over.

  2. Nothing counts folds per nudge. The only state tied to a nudge is the requireNudge credit: NUDGE_CREDIT_ROUNDS = 20 model rounds, issued each time a nudge reaches the model. The docs state that "Successful folds do NOT consume the credit". requireRecommendation judges coverage only, and "subdividing a recommended span into smaller chunks passes". So even with both gates on, one nudge allows any number of folds above the character floor for 20 rounds.

  3. bili's own fold producer already has both guards this path lacks. The [preflight] overflow / auto-fold loop commits at most one serial fold per round outside an emergency:

    const maxFoldsPerRound = emergencyFold ? Infinity : 1;
    for (let oi2 = 0; oi2 < ordered.length && appliedThisRound < maxFoldsPerRound && !poolQueueDrop; oi2++) {
      // … commitFold(…)
    }

    The external-summary batch path just above it commits up to MAX_PREFLIGHT_SUMMARY_BATCH = 4 nominations together, in the same request. Auto-fold is also benefit-gated by AUTO_FOLD_MIN_BENEFIT_TOKENS = 4096 (src/external-summary-settings.ts): once before it arms, and again after the fold, where a fold under the bar triggers the backoff (auto-fold sharp edges: a sub-floor autoFoldTargetTokens causes a fold storm, and externalSummary.* file edits do not take effect until a PUT /__bili/config #3444). This loop never runs for compress calls. maxFoldsPerRound is quoted here as an example of the guard this path lacks, not as the cause.

  4. bili's own prompt already describes the cost, but only as advice. FIRST_SIGHT_DRAIN_NOTE says: "Do NOT fold piecemeal across turns: every small front fold re-bills the whole remaining tail to the prefix cache, while one batched call lands once." It is only appended to first-sight backlog nudges, and no code enforces it.

Mechanism. Each accepted call replaces ctx.session.state immediately. Typically the model reads the receipt and then folds the next sub-range, so the next call arrives in a later upstream request. That request's history differs from the previous one starting at the new fold point, so the prefix cache misses from there to the end. The per-fold attribution (fold-attribution X(firstFoldStartTokens), #3307) and the Post-fold rebuild ledger line measure exactly this.

Suggested fix

Both parts are config-first and touch only applyRanges. bili's own overflow and emergency folding is unchanged.

Part 1: minimum benefit for model-initiated folds (small; do this first).

  • New key compress.minFoldBenefitTokens (integer ≥ 0). Proposed default: AUTO_FOLD_MIN_BENEFIT_TOKENS (4096), so both fold producers share one bar. 0 restores today's behaviour. If you prefer to keep 4096 internal, as decided in auto-fold sharp edges: a sub-floor autoFoldTargetTokens causes a fold storm, and externalSummary.* file edits do not take effect until a PUT /__bili/config #3444, apply the constant without a new key.

  • Implement it by generalising the existing refusal:

    // applyRanges: today the refusal fires only when netDelta <= 0
    const floor = inPressureBandOrEmergency(ctx) ? 1 : Math.max(1, minFoldBenefitTokens(ctx.config));
    if (r.blocksCreated > 0 && touchedBlocks > 0 && netDelta < floor) {
      return compressResult(belowFloorReceipt(netDelta, floor), "refused", 0,
        netDelta <= 0 ? "gate:no-net-shrink" : "gate:below-min-benefit");
    }
  • Exemptions: at or above maxContextLimit, or in an emergency, the floor drops to 1, so a fold the session needs to survive is never blocked. Refold-only calls (summary updates) keep today's check.

  • Return an error receipt, not a silent skip, for three reasons:

    • A silent skip tells the model its range is folded when it is not, so it may drop details it believes are summarised.
    • The receipt is the only channel that teaches the model to batch.
    • The refusal plumbing already exists (recordCompressFailure, the loop breaker, one-line collapse on repeats), so refusals cannot turn into a retry storm.

    Suggested text: [Compression refused: this call would reclaim ~2,812 tokens, below compress.minFoldBenefitTokens (4,096). Every fold re-bills the prompt-cache tail. Put the other listed ranges into ONE compress call, or continue the task. Nothing was folded and the cache is untouched.]

Part 2: per-nudge fold budget.

Impact

  • Folds per nudge: 3.8 on average (38 nudge-follow folds over 10 nudges), 5 at most. All happened below the pressure band (32.8% usage, band at 70%).
  • Re-bill per fold: about 15.9K uncached tokens (619K post-fold rebuild ÷ 39 folds).
  • Share of tokens: post-fold rebuild is about 11% of the session's 5.7M uncached tokens and about 0.19% of its 319.5M input tokens. It is a secondary cost (the session still hit 98.2% cache), but it is the part bili controls directly.
  • Avoidable share (estimate, not a measurement): one fold event per nudge would give 11 events instead of 39. At the session average, that avoids about 28 × 15.9K ≈ 445K uncached tokens, roughly 72% of the rebuild line. The real figure depends on where each batched fold starts, because a batched fold re-bills from its earliest range.
  • Small folds: a 3.2K fold costs about 15.9K uncached tokens once, about 5× what it removes from each later request. With a cached/uncached price ratio d, it pays back only after 15.9K ÷ (3.2K × d) later requests: about 50 at d = 0.1. This is the same fixed-cost argument that set the 4096-token gate for auto-fold in auto-fold sharp edges: a sub-floor autoFoldTargetTokens causes a fold storm, and externalSummary.* file edits do not take effect until a PUT /__bili/config #3444.
  • Scaling: the growth step (nudgeGrowthTokens) is flat by design and independent of window size. A large-window session therefore spends more of its life below the pressure band, where every fold is optional and this pattern repeats.

Known-issue cross-references

Related issues, not proof:


标题

compress:一次 nudge 可能引发一连串模型主动发起的小折叠,而每次折叠都会让提示缓存的尾部重新计费

环境

  • billion-context 0.1.216(npm 包 billion-context,CLI bili;dist/build-info.json 中的 commit 为 6a641f378)。
  • Claude Code 经 bili 的 /bili/ 改写通道接入 OpenRouter 托管的 DeepSeek 模型(deepseek-v4.1-flash),bili 解析出的上下文窗口约 968K token。
  • 已启用 T1/T2/T3 分层压缩,compress.maxContextLimit = 70%,其余 compress.* 均为默认值。
  • 下文统计的 39 次折叠全部来自模型调用 compress;按 bili 的分类,其中 38 次为 nudge 折叠,1 次为主动压缩。

摘要

在一次包含 1,632 个请求的生产会话中,10 轮 nudge 共引发 39 次模型主动折叠,平均每轮 3.8 次;最多的一轮有 5 次,全部发生在同一分钟内。其中两次只回收了约 2.8K 和 3.2K token,而触发它们的那次 nudge 推荐量约为 120K。

被接受的折叠都会立即提交。折叠分散在相邻的几个上游请求里时,每个请求都在新的位置改写历史,提供方的前缀缓存从该位置到末尾全部失效。bili 自己的缓存账本把 619K 未命中 token(平均每次折叠约 15.9K)记在"折叠后重建"名下。

bili 对自己发起的折叠早已设了防线:preflight 循环在非紧急情况下每轮只串行提交一次折叠,auto-fold 有 4096 token 的收益门槛(#3444),nudge 文案也明确要求模型不要零碎折叠。模型主动的 compress 路径却一条都没有执行:

  • 同一次 nudge 之后,折叠多少次都接受。
  • 净收缩只要不少于 1 token 就接受;另外只有一个约 5,000 字符的 minCompressRangeChars 下限。

即使打开 requireNudge 和 requireRecommendation,这两点也都不会改变。

复现步骤

任何支持工具调用的模型都可以。要验证的是 bili 会接受什么,而不是模型会怎么选,所以第 4 步直接要求模型拆分。

  1. 用隔离的配置在临时端口上启动一个实例:

    export XDG_CONFIG_HOME="$(mktemp -d)"
    mkdir -p "$XDG_CONFIG_HOME/billion-context"
    cat > "$XDG_CONFIG_HOME/billion-context/billion-context.json" <<'EOF'
    {
      "compress": {
        "modelContextLimit": 200000,
        "nudgeGrowthTokens": 20000,
        "requireNudge": true,
        "requireRecommendation": true
      }
    }
    EOF
    npx -y -p billion-context@0.1.216 bili start --host 127.0.0.1 --port 18787

    这里故意打开 requireNudge 和 requireRecommendation,用来说明现有最严格的两道门控同样会放行这种连续折叠。

  2. 在另一个终端里让 Claude Code 走这条通道:

    ANTHROPIC_BASE_URL="http://127.0.0.1:18787/bili/<anthropic-compatible-upstream-base-url>" claude

    如果这台机器装了 bili 的 Claude 插件,插件的 SessionStart hook 会在每次会话开始时重新指定 base URL,请换一个没有装插件的环境或用户配置。

  3. 让上下文增长越过 nudge 步长:让 agent 完整读取三四个大文本文件(例如本仓库的 docs/configuration.md 和 README.md),直到模型收到压缩 nudge。此时 acp_status 会列出可压缩范围。

  4. 发送:"把 acp_status 列出的最大的那个可压缩范围拆成四个相邻的子范围,分四次 compress 调用来折叠,每次只折叠一个子范围,等上一次的回执回来再发下一次。"

  5. 观察代理日志,以及 http://127.0.0.1:18787/__bili/ 面板里的 Cache economics 和 compression recovery (per nudge round) 两个视图。

对照组: 新开一个会话重复第 1–3 步,然后要求用一次 compress 调用、同时带上这四个子范围来折叠同一段内容。

预期与实际

场景 预期 实际(0.1.216)
一次 nudge 之后分四次调用 第一次调用正常折叠;同一 nudge 窗口内的后续调用被拒绝(或延后),回执提示模型合并成一次调用;上下文已进入压力区间时不受此限 四次全部接受:日志里有四条 [acp-proxy: compress requested 1 range(s): …],模型收到四个 [Compressed … → 1 block(s), ~N tokens saved.] 回执,没有任何 REFUSED
上游提示缓存 这次 nudge 只产生一次折叠后重建,与对照组相同 产生四次重建:每次折叠之后的请求,缓存命中 token 都会下降,"折叠后重建"每次折叠都会增加一笔
压力区间以下、只回收约 3K token 的折叠 以错误回执拒绝,状态不提交,缓存不动(与现有 gate:no-net-shrink 的拒绝形式相同) 被接受;只检查 netDelta <= 0 和约 5,000 字符的 minCompressRangeChars
打开 requireNudge + requireRecommendation 不适用:这两道门控管的是时机和范围,不管次数和大小 不起作用:nudge 信用是一个 20 轮的窗口,成功折叠不消耗信用;把推荐范围再拆小也被明确允许

证据

数据来自一次包含 1,632 个请求的生产会话,全部取自 bili 自带的会话报告(Cache economics 与 compression recovery (per nudge round));标注"推算"的行由前述数据计算得出。

指标 数值
输入 token 319.5M,其中 313.8M 命中缓存(98.2%)
nudge 轮数 10
折叠次数 39(38 次 nudge 折叠,1 次主动压缩)
每轮 nudge 的折叠次数(推算) 平均 3.8(38 / 10),最多 5
回收率(Σ 折叠量 ÷ Σ 推荐量) 110.5%
最严重的一次连发 一次 nudge 后一分钟内连续 5 次折叠;其中两次分别只回收 ≈2.8K 和 ≈3.2K token,而该 nudge 的推荐量 ≈120K
折叠时的上下文占用 317,398 / 968,000 = 32.8%(压力区间从 70% 开始)
未命中缓存的输入 token 5.7M = 新内容 1.9M + 折叠后重建 619K + 未归因回补 3.2M
每次折叠的重建量(推算) 619K / 39 ≈ 15.9K token

根因

以下代码引用均指 0.1.216 tarball 中的 dist/index.js,可直接搜索文中引用的标识符。

  1. 模型主动的折叠只检查收益是否为零。 compress 的处理函数是 applyRanges。解析完参数、走完可选的 requireNudge / requireRecommendation 之后,唯一的收益检查如下(据打包代码整理格式):

    if (r.blocksCreated > 0 && touchedBlocks > 0 && netDelta <= 0) {
      // … "Compression refused: no net shrink …"
      return compressResult(receipt, "refused", 0, "gate:no-net-shrink");
    }
    ctx.session.state = res.state;   // 提交

    哪怕只让上下文缩小 1 个 token,折叠也会被提交。唯一的尺寸下限是 kernel 的 compress.minCompressRangeChars(旧名 minCompressRange),默认约 5,000 字符,英文约合 1.2K token。它按整次调用的总量计算,重新折叠已有 block 的范围也不受它约束。一次 2.8K token 的折叠就超过这个下限两倍多。

  2. 没有任何机制按 nudge 统计折叠次数。 唯一与 nudge 绑定的状态是 requireNudge 的信用:NUDGE_CREDIT_ROUNDS = 20 个模型轮,每次 nudge 真正送达模型时发放。文档写明 "Successful folds do NOT consume the credit"(成功的折叠不消耗信用)。requireRecommendation 只按覆盖范围判断,并且 "subdividing a recommended span into smaller chunks passes"(把推荐范围拆成更小的块可以通过)。所以即使两道门控都打开,一次 nudge 仍允许在 20 轮内做任意多次折叠,只要每次超过字符下限即可。

  3. bili 自己的折叠路径恰好具备这条路径缺少的两道防线。 [preflight] 溢出兜底 / auto-fold 循环在非紧急情况下每轮最多串行提交一次折叠:

    const maxFoldsPerRound = emergencyFold ? Infinity : 1;
    for (let oi2 = 0; oi2 < ordered.length && appliedThisRound < maxFoldsPerRound && !poolQueueDrop; oi2++) {
      // … commitFold(…)
    }

    紧挨着它上方的外部摘要批处理路径最多把 MAX_PREFLIGHT_SUMMARY_BATCH = 4 个候选放在同一个请求里一起提交。auto-fold 还受 AUTO_FOLD_MIN_BENEFIT_TOKENS = 4096(src/external-summary-settings.ts)约束:武装前检查一次,折叠后再检查一次,不达标就进入退避(auto-fold sharp edges: a sub-floor autoFoldTargetTokens causes a fold storm, and externalSummary.* file edits do not take effect until a PUT /__bili/config #3444)。这个循环从不处理 compress 调用。这里引用 maxFoldsPerRound,是为了说明这条路径缺的正是这种防线,并不是说问题出在它身上。

  4. bili 自己的提示词已经写明这笔成本,但只停留在建议。 FIRST_SIGHT_DRAIN_NOTE 写道:"Do NOT fold piecemeal across turns: every small front fold re-bills the whole remaining tail to the prefix cache, while one batched call lands once."(不要跨轮零碎折叠:每次小的前部折叠都会让剩余尾部在前缀缓存里重新计费,而一次批量调用只落地一次。)这段话只在首次接手积压时才附加,也没有任何代码强制执行。

机制。 每次被接受的调用都会立即替换 ctx.session.state。模型通常读完回执才折叠下一个子范围,所以下一次调用出现在后一个上游请求里。这个请求的历史与上一个请求从新的折叠点开始不同,前缀缓存从这里到末尾全部失效。逐次折叠的缓存归因(fold-attribution X(firstFoldStartTokens),#3307)和账本中的"折叠后重建"一项,衡量的正是这部分成本。

修复建议

两部分都以配置为先,且只改动 applyRanges;bili 自身的溢出兜底和紧急折叠保持不变。

第一部分:给模型主动的折叠加最低收益门槛(改动小,建议先做)。

  • 新增配置项 compress.minFoldBenefitTokens(整数,≥ 0)。建议默认值取 AUTO_FOLD_MIN_BENEFIT_TOKENS(4096),让两类折叠用同一条线;设为 0 即恢复现状。如果维护者倾向于像 auto-fold sharp edges: a sub-floor autoFoldTargetTokens causes a fold storm, and externalSummary.* file edits do not take effect until a PUT /__bili/config #3444 那样把 4096 保留为内部常量,也可以不加配置项,直接套用该常量。

  • 在现有的拒绝逻辑上推广即可:

    // applyRanges:目前只在 netDelta <= 0 时拒绝
    const floor = inPressureBandOrEmergency(ctx) ? 1 : Math.max(1, minFoldBenefitTokens(ctx.config));
    if (r.blocksCreated > 0 && touchedBlocks > 0 && netDelta < floor) {
      return compressResult(belowFloorReceipt(netDelta, floor), "refused", 0,
        netDelta <= 0 ? "gate:no-net-shrink" : "gate:below-min-benefit");
    }
  • 豁免:占用达到 maxContextLimit 或处于紧急状态时门槛降为 1,保证会话赖以存活的折叠永远不会被拦下;只更新已有 block 摘要的重新折叠仍沿用现有检查。

  • 返回错误回执,而不是静默跳过,理由有三:

    • 静默跳过会让模型以为范围已经折叠,从而丢掉它以为已被摘要的细节。
    • 回执是让模型学会合并调用的唯一渠道。
    • 拒绝所需的机制都已存在(recordCompressFailure、循环熔断、重复拒绝折叠成一行),不会引发重试风暴。

    建议文案:[Compression refused: this call would reclaim ~2,812 tokens, below compress.minFoldBenefitTokens (4,096). Every fold re-bills the prompt-cache tail. Put the other listed ranges into ONE compress call, or continue the task. Nothing was folded and the cache is untouched.]

第二部分:每次 nudge 的折叠预算。

影响

  • 每轮 nudge 的折叠次数: 平均 3.8 次(10 轮 nudge 共 38 次 nudge 折叠),最多 5 次;全部发生在压力区间以下(占用 32.8%,压力区间从 70% 开始)。
  • 每次折叠的重新计费: 约 15.9K 未命中 token(619K 折叠后重建 ÷ 39 次折叠)。
  • 占比: 折叠后重建约占本会话 5.7M 未命中 token 的 11%,约占全部 319.5M 输入 token 的 0.19%。这是次要成本(会话缓存命中率仍有 98.2%),但却是 bili 能直接控制的那一部分。
  • 可避免的部分(估算,非实测): 如果每轮 nudge 只产生一次折叠,折叠次数会从 39 次降到 11 次。按会话平均值计算,可少付约 28 × 15.9K ≈ 445K 未命中 token,约为重建总量的 72%。实际数字取决于每次批量折叠的起点,因为批量折叠要从其中最靠前的范围开始重新计费。
  • 小折叠的账: 一次 3.2K 的折叠先要付出约 15.9K 未命中 token,约是它从此后每个请求中省下的量的 5 倍。设缓存命中价与未命中价之比为 d,它要经过 15.9K ÷ (3.2K × d) 个后续请求才能回本:d = 0.1 时约 50 个。auto-fold sharp edges: a sub-floor autoFoldTargetTokens causes a fold storm, and externalSummary.* file edits do not take effect until a PUT /__bili/config #3444 为 auto-fold 设 4096 token 门槛,依据的正是同样的固定成本逻辑。
  • 规模效应: 增长步长(nudgeGrowthTokens)按设计是固定值,与窗口大小无关。窗口越大,会话在压力区间以下停留的时间越长;而在这个区间里每次折叠都是可选的,这种模式会反复出现。

相关已知问题

以下列为相关问题,而非佐证:

Activity

  1. changed the title [-]`compress`:一次 nudge 可能引发一连串模型主动发起的小折叠,而每次折叠都会让提示缓存的尾部重新计费[/-] [+]`compress`: A single nudge may trigger a chain of model-initiated minor folds, and each fold causes the tail of the prom | `compress`:一次 nudge 可能引发一连串模型主动发起的小折叠,而每次折叠都会让提示缓存的尾部重新计费[/+] on Nov 20, 2026
  2. ranxianglei commented on Nov 20, 2026

    @ranxianglei
    OwnerAuthor

    🤖 Powered by ework · qwen3.8-27b

    [bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

    查重 (dedup): Not a duplicate. Searched open+closed issues/PRs for the same fix (min-benefit floor for model-initiated folds, per-nudge fold budget, cache-tail re-billing). Only this issue (#7333) matches; minFoldBenefitTokens appears nowhere else. Related-but-distinct: #3444 (auto-fold 4096 benefit gate — a different fold producer), #5943 / #6760 / #6771 (nudge re-injection — the interaction Part 2 must compose with), #6720 (small per-fold yield — adjacent symptom).

    Labels: bug · p2 · claude-code · scope/kernel (cost/efficiency, not unavailability — the session still hit 98.2% cache). Note: the mirror label API is not implemented in the current ework shim (/issues/{n}/labels → "endpoint not implemented in ework shim"), so I could not apply them programmatically. Flagging for you to apply, or for the shim to gain the endpoint.

    Triage (verified against current master, not the 0.1.216 tarball):

    Root cause confirmed — model-initiated folds are committed after a zero-benefit check only:

    • applyRanges (src/stream.ts:1118, the single compress execution point across all lanes) has exactly one benefit test: the net-shrink gate at src/stream.ts:1459 — if (r.blocksCreated > 0 && touchedBlocks > 0 && netDelta <= 0) → gate:no-net-shrink. A fold that shrinks by 1 token is committed (commit at src/stream.ts:1477).
    • The only size floor is compress.minCompressRange (~5000 chars ≈ 1.2K tokens; counted across the whole call; does NOT apply to refolds of existing blocks). A 2.8K-token fold clears it 2× over.
    • Nothing counts folds per nudge. The only nudge-tied state is the requireNudge credit (NUDGE_CREDIT_ROUNDS = 20 model rounds, src/unprompted-gate.ts:23), minted when a nudge is actually injected; successful folds do NOT consume it. requireRecommendation judges coverage only ("subdividing a recommended span into smaller chunks passes").
    • bili's OWN fold producers already have the missing guards: the [preflight] overflow/auto-fold loop commits ≤1 serial fold per round outside emergency (maxFoldsPerRound), and auto-fold is benefit-gated by AUTO_FOLD_MIN_BENEFIT_TOKENS = 4_096 (src/external-summary-settings.ts:99, auto-fold sharp edges: a sub-floor autoFoldTargetTokens causes a fold storm, and externalSummary.* file edits do not take effect until a PUT /__bili/config #3444). The nudge text (FIRST_SIGHT_DRAIN_NOTE) says "Do NOT fold piecemeal across turns" but only as advice — no code enforces it.

    Layer: host-side gate gap in the compress execution path (src/stream.ts), not a kernel defect — the kernel's applyCompression faithfully executes what the host commits. The fix belongs in the host gate. No kernel/ change needed.

    Plan:

    Starting Part 1 now.

  3. ranxianglei commented on Nov 20, 2026

    @ranxianglei
    OwnerAuthor

    🤖 Powered by ework · qwen3.8-27b

    [bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

    🔶 需要决策

    背景:Part 1(最小收益下限)已完成,PR #7334 已开。Part 2(每次 nudge 的折叠预算)需要拍板,因为它与 #5943(requireNudge/requireRecommendation 不消耗 nudge credit)和 open PRs #6760/#6771(partial fold 后重新注入 T1)有交互。

    待决问题:是否引入 compress.foldsPerNudge(int ≥ 0,unset/0 = 不限制 = 当前行为)?

    选项:

    不决定的默认行为:只合入 Part 1(PR #7334),Part 2 留待后续。

  4. ranxianglei commented on Nov 20, 2026

    @ranxianglei
    OwnerAuthor

    🤖 Powered by ework · qwen3.8-27b

    [bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

    结构化八问

    1. 是什么问题:一次 nudge 可能引发一连串模型主动发起的小折叠(每次只回收 2-3K tokens),而每次折叠都会重写上下文前缀并重新计费提示缓存尾部,导致大量 uncached tokens。

    2. 什么原因导致的:applyRanges(src/stream.ts:1459)只有一个零收益检查(netDelta <= 0),没有最小收益下限。模型可以在一次 nudge 后发起多次小折叠,每次都只回收少量 tokens,但每次都会触发缓存尾部重新计费。

    3. 影响范围与严重程度:影响所有使用 model-initiated compress 的会话。严重程度:性能问题(缓存尾部重复计费),不是功能问题。issue 中报告的 968K 窗口会话:一次 nudge 后 39 次小折叠导致 619K uncached tokens(~15.9K/fold)。

    4. 怎么修复的:在 applyRanges 的 手动 compress 仍接受零/负净收益折叠,先改写缓存再提示上下文增长 #2748 零收益检查上增加最小收益下限(compress.minFoldBenefitTokens,默认 4096 = AUTO_FOLD_MIN_BENEFIT_TOKENS,与 auto-fold 路径 auto-fold sharp edges: a sub-floor autoFoldTargetTokens causes a fold storm, and externalSummary.* file edits do not take effect until a PUT /__bili/config #3444 共用同一门槛)。低于下限的折叠在压力带以下被拒绝,返回 gate:below-min-benefit + 教育模型批量折叠的收据。压力带/紧急状态下限降为 1(生存折叠不被阻止)。preflight 车道(loopTracking:false)豁免(bili 自己的溢出折叠不受限)。

    5. 是否有回归问题:4 个现有测试文件(issue2360, issue847, issue2146, issue5683)的折叠大小低于 4096 下限,已扩大消息大小(5000→10000 chars)修复。loop-compress.test.ts 有 6 个预先存在的失败(git stash 验证:master 上无此 PR 也失败),不是本 PR 导致的。

    6. 是否破坏缓存:不破坏。本 PR 的目的是减少缓存尾部的重复计费。no-net-shrink 收据(netDelta <= 0)字节级不变。

    7. 是否改了设计方向:没有。本 PR 在现有的 手动 compress 仍接受零/负净收益折叠,先改写缓存再提示上下文增长 #2748 零收益检查上增加最小收益下限,不改变架构或数据流。

    8. 是否引入了新的配置:是。新增 compress.minFoldBenefitTokens(int ≥ 0,默认 4096,0 = 传统行为 = 任何正收益都提交)。host-side only,不映射到 kernel nudge.* 字段。

    交付

    合并判定:Part 1 可合并(host-side only,无 kernel 变更,有回归测试,CI 未跑)。Part 2 等决策。

  5. ranxianglei commented on Nov 20, 2026

    @ranxianglei
    OwnerAuthor

    🤖 Powered by ework · qwen3.8-27b

    [bot] 🏷 (ai: vllm-qwen/qwen3.8-27b)

    PR #7334 CI 回归修复进展

    CI 红了 32 个测试——全是我 Part-1 的 min-benefit floor gate 误伤了其他功能的测试(小 fixture 的 fold 被 4096 floor 拒了)。

    已修:

    1. 12 个测试文件加了 minFoldBenefitTokens: 0(bridge e2e、imageRef、fork adoption、dsh、resume、model switch、tag loop 等)
    2. minFoldBenefitTokens() helper 补了 config.compress?.minFoldBenefitTokens fallback(tests 用 defaultConfig() 直接绕过了 resolveCompress)

    本地验证: 36/39 通过。剩 3 个 resume-ref-inheritance.test.ts 还在挂("parent must have folded a block (got 0)"),config 里已经设了 minFoldBenefitTokens: 0 但 fold 仍被拒——正在查 resolveRequestConfig 的 config 传递链。

    修完这 3 个就 push,CI 重跑。Part 2 (foldsPerNudge) 决策请求不变,等你拍板。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions