Skip to content

plugin mode: no hard backstop when the agent ignores the compress nudge — context grows past the window and upstream 400s #453

Closed

Description

@ranxianglei

Found while investigating #14 ("bili总是循环").

Repro (real session)

bili pi (plugin mode, OpenAI chat-completions wire) with qwen3.8-27b on vLLM (262144 window), bili.log for session 01a06275:

  • 14:14:00 nudge INJECT T1 (pendingT1 52017 ≥ 50000) — model does not call compress
  • 14:14–14:19 nudge idle ... blocked: T1 (cadence) while usage climbs 56% → 70%
  • 14:19:34–14:20:49 nudge INJECT T1 reason="OVER-LIMIT T1" usage 77% → 85% — still no compress
  • 14:27:39 upstream rejects: 400 Requested token count exceeds the model's maximum context length of 262144 tokens. You requested a total of 262817 tokens: 131745 tokens from the input messages and 131072 tokens for the completion.

The client (pi) then falls back to its own native compaction → the model loses all working state and re-derives its analysis from scratch (compounding the perceived "loop" in #14). A second session (01a062a7, qwen3.8-flash) sat at 80–85% usage with nudge INJECT T1 firing every turn — the model never called compress at all.

Impact

In plugin mode the nudge is advisory: bili trusts the ACP-native agent to execute compress. When the model simply ignores the injected nudge (weaker instruction-following, e.g. qwen3.8-27b), nothing stops the payload from overflowing the window. Proxy mode has preflight/preflightCompressIfNeeded as the backstop; plugin mode has none.

Suggested fix (discussion)

Options, roughly in ascending invasiveness:

  1. Clamp the outgoing output budget: max_tokens/max_completion_tokens = min(requested, window − estimated input − margin) so input+output can never exceed the window on request-rebuilding upstreams (vLLM rejects instead of clamping). Cheap, no wire changes; doesn't fix input-overflow itself.
  2. Proxy-side emergency truncation: when estimated input tokens exceed the window and the agent hasn't compressed after N over-limit nudges, drop/truncate the oldest inessential history server-side for that request (like proxy-mode preflight) even though the session is plugin-mode. Needs care to keep the agent's view consistent.
  3. Escalating nudge: make the over-limit nudge increasingly forceful / repeat every turn once usage > 85% instead of cadence-blocking (the cadence block kept nudges silent for 5 minutes in the repro while usage grew 56%→70%).

(3) is probably the right first step; (2) needs a design pass.

Environment

  • bili 0.1.63 (dev build of master@7135709), plugin mode via bili pi, upstream vLLM private host, MITM tunnel

Activity

  1. ranxianglei commented on Sep 2, 2026

    @ranxianglei
    OwnerAuthor

    🤖 ework agent · qwen3.8-27b

    Traced this through the code (master@7135709) rather than taking the log at face value. The diagnosis holds, but the root cause sits one layer deeper than the framing — and it changes which fix is the real backstop.

    What I verified

    • Plugin mode delegates all compression to the agent. Nudge injection is gated !pluginMode (src/server.ts:1452/1556/1721) and the response passes through verbatim (src/server.ts:2538). The proxy computes the nudge decision, exposes it via /__bili/plugin/status, then waits for the agent to call compress. If the model ignores it, nothing the proxy does folds history server-side.
    • The server-side backstop exists in BOTH modes. preflightCompressIfNeeded is called unconditionally (src/server.ts:1268), not gated on pluginMode. So "plugin mode has none" isn't quite right — it has preflight. The problem is preflight's trigger, below.
    • There is no clamp on the outgoing max_tokens. The client's value passes through untouched; the only max_tokens writes in src/ are preflight's internal 8192-budgeted summarization sub-calls (src/preflight.ts:143-148).

    Root cause — this is an input+OUTPUT overflow, not input-alone

    The 400 is 131745 (input) + 131072 (completion) = 262817 > 262144. Input alone (131745) is well under the window. Two things conspire:

    1. Preflight's trigger is tokenCount = max(lastInputTokens, payloadEstimate) >= limit (src/server.ts:2179), where payloadEstimate = estimateCoreMessages(...) counts message text only — it excludes the system prompt + tool definitions (src/preflight.ts:90-94). So the trigger undershoots the true input by the fixed system+tools overhead (~10–20k here).
    2. On non-Anthropic wires the effective limit is already reduced by the output-headroom reservation to window − maxOutput (src/server.ts:1231-1246) — exactly the overflow boundary. Preflight is meant to fire the moment input crosses window − maxOutput, but because the estimate undershoots, the true input crosses the line while the trigger is still just below it → no preflight → the request goes out → vLLM rejects input+output.

    In plugin mode this is maximally exposed: the reactive nudge/truncate path is handed to the agent, and qwen3.8-27b never calls compress, so preflight is the only thing between the session and the 400 — and it misses by the undershoot margin. The cadence block (56→70%, silent) is a secondary contributor: it kept the advisory nudge quiet during the fastest growth phase.

    Assessment of the three options

    • (1) Clamp the outgoing output budget — max_tokens = min(requested, window − estimated_input − margin). This is the actual hard backstop for the input+output class and the only option that makes the overflow impossible on request-rebuilding upstreams, because the proxy controls both sides of the sum. Fixes the exact repro. Applies to both modes. Residual risk only if the input estimate undershoots more than the margin — close that by counting system+tools into the estimate (or a conservative margin). Anthropic stays exempt (its input limit is enforced independently of max_tokens).
    • (3) Escalating nudge — closes the silent cadence-gap and raises compliance odds, but does not guarantee: the repro shows per-turn OVER-LIMIT T1 nudges at 77–85% were still ignored. A mitigation, not a backstop.
    • (2) Proxy-side emergency truncation in plugin mode — the definitive backstop, but the consistency hazard is real and larger than noted: the plugin-mode summary carrier is the agent's own compress tool call, whereas preflight builds tool-call-less blocks whose acp_summary carrier is re-voiced as a user message via systemToUser (a proxy-mode semantic). Folding behind the agent's back risks it re-sending unfolded ranges next turn, and preflight's carrier handling was never validated for plugin mode. Genuinely needs the design pass you flagged.

    Recommendation (staged, complementary — not competing)

    1. Ship (1) first — small, pure-safety (can only ever lower max_tokens), both modes, kills the reported 400 outright.
    2. Ship (3) alongside — stop cadence-blocking above the emergency threshold so weak models get per-turn pressure instead of a 5-min silence.
    3. File (2) as a separate follow-up issue for the design pass (agent-view consistency + carrier semantics).

    Happy to implement (1)+(3) now and open a PR — confirm the staging (or tell me to reorder/split) and I'll go.

  2. ranxianglei commented on Sep 2, 2026

    @ranxianglei
    OwnerAuthor

    在当前 master(ac4fbcb,已含 #455/#456/#452)上逐条复核了楼上的分析,全部属实,补充精确代码位置:

    (a) preflight 双模式无条件运行 ✅ server.ts:1273(在 !countTokens && !responsesCompact 内,不受 pluginMode 门控)。

    (b) 转发请求无 max_tokens 钳制 ✅ 全代码库唯一设置 max_tokens 的地方是 preflight 内部子调用(src/preflight.ts summaryPayload,固定 MAX_SUMMARY_OUTPUT_TOKENS);主请求 body 原样透传。server.ts:1236-1250 的 reserveOutputHeadroom 只降低内部波段窗口(reqConfig.modelContextLimit = window − maxOutput),它影响 nudge/truncate 决策,不影响实际转发出去的请求。

    (c) 触发下冲 → input+output 400 ✅ server.ts:2199-2200:
    tokenCount = max(session.stats.lastInputTokens, estimateCoreMessages(prepared.processedMessages))
    而 src/preflight.ts:90-95 的 estimateCoreMessages 只累加消息文本(defaultCountTokens(m.text)),不含 system+tools(约 10-20k)。复现数字完全对上:真实 input 131745 > 保留限 131072,但 payloadEstimate ≈ 117k、lastInputTokens 又滞后一轮 → 双双低于限 → preflight 不触发 → 131745 + 131072 = 262817 > 262144 → vLLM 直接 400(拒绝而非截断)。

    次因 cadence 消音可完全 host 侧修复:NudgeDecision 携带 contextUsage/reason/tier,且 renderNudgeText(decision, prompts) 不依赖 shouldInject(server.ts:1447/1547/1707 三处注入点都是 host 侧判断)→ 紧急升级(usage ≥ 阈值时绕过 cadence)不需要动 acp-kernel。

    过时前提:本 issue 开头描述的 "nudge 被 !pluginMode 门控" 是 0.1.69 的状态,#456(v0.1.70)已恢复 plugin 模式 nudge 注入。当前剩余问题 = 触发下冲 + 无 max_tokens 钳制 + cadence 消音三件事。

    建议按楼上分级落地:

    1. max_tokens 钳制(硬兜底,先发):非 Anthropic wire(Anthropic 输入预算独立,见 util.ts:222 shouldReserveOutputHeadroom),在 prepare* 重建 body 时 max_tokens = min(请求值, window − 有效输入估算 − margin)。这是唯一让 input+output 溢出不可能发生的选项,双模式受益。
    2. 紧急升级 nudge(缓解):host 侧在 usage ≥ 紧急阈值且被 cadence 消音时强制注入,改 server.ts 三处判断,零 kernel 改动。
    3. plugin 模式代理侧截断 → 另开 issue 单独跟踪(载体语义风险:preflight 构建无 tool-call 的块 + acp_summary 变 user 载体 = proxy 模式语义,agent 可能重发未折叠区间)。

    维护者确认分级后我来实现 1+2。

  3. ranxianglei commented on Sep 2, 2026

    @ranxianglei
    OwnerAuthor

    🤖 ework agent · qwen3.8-27b

    [bot] 🏷 已按确认的分级实现 1 + 2,PR: #467 (基于 ac4fbcb,未动 version)。

    1. max_tokens 钳制(硬兜底) — prepareOpenai/prepareResponses 重建 body 时把 max_tokens / max_completion_tokens / max_output_tokens 压到 window − 输入估算 − margin。estimateInputTokens() 取「上轮真实 input」与「消息 + system + tools 现算」的较大值,补上了 preflight 估算漏掉 system/tools 的下冲。Anthropic 豁免(输入预算独立)。cap 低于地板(输入本身快撑满窗口)时不动,交给 preflight/self-heal——不饿死输出。这是唯一让 input+output 溢出不可能发生的一层,双模式受益。

    2. 紧急升级 nudge(缓解) — host 侧在 usage ≥ 阈值且被 kernel cadence/growth 消音时强制注入,三处判断各 OR 一个 emergencyNudge(),零 kernel 改动;diagNudge 现在标 INJECT-ESC,不再谎报 idle。

    ⚠️ 一个需要你拍板的点:追了 kernel decideNudge,发现 pressure 分支(usage ≥ nudge.maxContextLimitPct,默认 0.75)没有 cadence 门——kernel 在那之上本来就每轮注入。所以唯一被 cadence 消音的区间在 0.75 之下;issue 里说的「≥85%」落在 kernel 已在注入的区间里,是空操作。因此我把升级线设在 0.70(压在 0.75 正下方,补 pre-limit 静默爬升那段),是单个命名常量,改起来一行。想要别的值、或做成三级 compress 可配,说一声即调。

    测试:新增 tests/issue453-backstop.test.ts——三个 helper 单测(clamp 触发/适配/floor;estimateInputTokens 真实值主导 + turn1 system+tools;emergencyNudge armed/线/自定义/空-range guard)+ 两个真实 proxy+mock upstream 集成测试(clamp 在大 system 把真实输入顶过保留窗口而 preflight 跳过时确实压低转发 max_tokens,复现本 issue 下冲;escalation 在 72% 恰好多一条尾随 nudge、60% 无)。typecheck ✅ · 全量 924/925(唯一失败是 launcher.test.ts 在本机解析到已安装的 codex 二进制,属环境性——stash 掉我的改动后同样失败)· build ✅ · e2e preflight ✅(完整 e2e 需活的 Responses upstream → CI/手动)。

    option 3(plugin 模式代理侧截断)按你的计划未包含——载体语义风险(preflight 建无 tool-call 块 + acp_summary 变 user = proxy 语义,agent 可能重发未折叠区间),留给你另开 issue 单独跟踪。

    等你 review / merge。

  4. added a commit that references this issue on Sep 3, 2026
  5. ranxianglei commented on Sep 3, 2026

    @ranxianglei
    OwnerAuthor

    Fixed by #467 (merged c2f0229): tier-1 max_tokens clamp (clampOutputBudget/estimateInputTokens, closes the input+output>window 400) + tier-2 host-side emergency nudge at 0.7. Tier-3 (proxy-side emergency truncation) remains tracked in #470 as intentionally deferred. Closing this as fixed — remaining work lives in #470.

  6. added 2 commits that reference this issue on Sep 3, 2026
  7. added a commit that references this issue on Oct 4, 2026
  8. added a commit that references this issue on Oct 19, 2026
  9. added a commit that references this issue on Oct 24, 2026
  10. added a commit that references this issue on Oct 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions