Skip to content

feat(compress): opt-in nudge-low-effort — clamp thinking/effort on first compression-nudge turn - #1780

Open
ranxianglei wants to merge 1 commit into
masterfrom
2026-09-30_nudge-low-effort
Open

ranxianglei wants to merge 1 commit into
masterfrom
2026-09-30_nudge-low-effort

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Adds an opt-in compress.nudgeLowEffort (boolean, default off). When enabled, on the first response of a compression-nudge episode bili clamps the outbound request's existing effort/thinking fields down to the wire floors, then restores on the next request (one-shot). Reuses the existing outputSteering lowerEffort clamp infra.

Motivation (#853 / #1640): on the turn where the model writes a compression summary, thinking models burn ~9.5k reasoning tokens bounded only by the output budget. Cutting that one turn's effort is the cheapest "save money on compression" lever — no bypass model, no wire-shape change beyond the effort param.

How it works

  • The four prepare paths record whether a fresh compression nudge was injected this turn (Prepared.injectedNudge).
  • At the forward boundary, if nudgeLowEffort === true and this is the first turn of a nudge episode (tracked via session.metadata.nudgeLowEffortPrev, refreshed every pass), the outbound body's effort fields are clamped via applyNudgeEffortClamp: Anthropic thinking.budget_tokens -> floor 1024, Gemini thinkingConfig.thinkingBudget -> floor 128, OpenAI/Responses reasoning_effort/reasoning.effort -> low.
  • Only lowers, never raises, never injects absent fields -> cannot bust the prompt cache (effort is a request param outside the message prefix). Independent of outputSteering.enabled.

Config surface

New field in the compress block (level 1 / global), in bili's own schema — NOT the kernel-owned outputSteering schema (avoids a cross-repo acp-kernel change; we reuse its already-existing lowerEffort internally).

Behavior change (default off = zero change)

  • Unset/false: byte-for-byte identical behavior — no regression path.
  • When true: one-shot reduction of thinking/effort on the first compression-nudge turn only. Tradeoff: fewer reasoning tokens (cost) vs. possibly reduced deferral judgment — the model has ~10% chance to defer compression, which is valuable; lowering effort may reduce it. Mitigation: a [nudge-low-effort] info log marks every clamp so operators can A/B the compress/defer distribution across runs.

Pre-flight (current master, acp-kernel 0.0.99)

  • npm run typecheck: clean
  • npm test: 3424 tests, 3405 pass, 0 fail, 19 skipped
  • npm run build: success
  • New tests: tests/nudge-low-effort.test.ts (per-wire clamp + floors, idempotency, safety on bad JSON / null protocol / array body, classifier-independence, one-shot episode decision cycle, config parse round-trip + non-boolean rejection).

Scope

Part of #1640 (split into two PRs by owner). This PR = low-thinking approach (first). Pure async compression is a separate follow-up PR (not here).

…rst compression-nudge turn

Add compress.nudgeLowEffort (boolean, default off): clamps the outbound request's existing effort/thinking fields to wire floors on the FIRST response of a compression-nudge episode, restoring on the next request (one-shot).

Reuses outputSteering lowerEffort clamp (Anthropic thinking.budget_tokens floor 1024 / Gemini thinkingBudget floor 128 / OpenAI+Responses reasoning_effort->low); only lowers, never raises/injects absent fields, so it cannot bust the prompt cache. Cuts thinking tokens spent writing the summary (#853 ~9.5k reasoning tokens). Default off = byte-for-byte unchanged behavior. Part of #1640.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 收到,开始按 PR Review 规范审查 #1780(nudge-low-effort)。初步状态:分支 2026-09-30_nudge-low-effort @ 49ca7577,单 commit、基于当前 master(30a23cb1)无落后;diff 共 6 文件 +238/-4;CI 各 job 目前 queued。接下来我会依次做:① 原 issue(#853/#1640)方向核对 → ② 回归清点(server.ts forward 边界 / config.ts strict-parse / output-steering.ts clamp 的既有行为)→ ③ 代码与测试有效性 → ④ diff 整洁度与热文件冲突。结论稍后回帖。

@github-actions

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-09-30_nudge-low-effort (49ca757)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.1747-pr.1780.3217f78

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr1780.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Review complete — mergeable (merge itself stays human-only).

① Original issue direction (#1640)

Matches the owner-approved mechanism exactly: floor 45 chose "压缩注入后的第一个回应直接降低思考参数,然后下一个请求立马恢复" (low implementation cost, no cache risk), floor 49 split into two PRs with this one first. Config placement in bili's own compress block (not kernel-owned outputSteering) avoids a cross-repo acp-kernel change while reusing the existing lowerEffort floors internally — the right boundary call, and the alternatives (kernel extension, side-model route) are correctly documented as rejected/deferred. One scope note for the record: the preflight server-side summary-call clamp discussed in #1640 is NOT in this PR — consistent with the owner's chosen nudge-turn mechanism; a possible follow-up if that path ever becomes a cost hotspot.

② Regression review — existing behaviors touched, and verdicts

The diff touches these load-bearing paths; each checked against pre-change code:

Existing behavior Verdict
Nudge injection decision (shouldInject / emergencyNudge gating, injectNudge config) Preserved. injectedNudge = willInjectNudge is recorded from the same boolean that gates the actual push (src/server.ts:3369/3603/3853/4147, pushes at 3429/3662/3888/4204); no early return between record and push, so flag and injection cannot desync.
Both compression modes Preserved / works in both. Nudge is proxy-injected in plugin mode too (#451 comment) — the flag rides on the same path, so dsh/plugin-mode sessions get the feature; nothing mode-specific was altered.
Output steering (outputSteering.enabled path in forward()) Preserved. New block sits after steering, before egress stamping; reads its own independent config key.
Egress stamp digest Correctly ordered. Clamp runs before stamping, so the digest covers the clamped bytes (no stale-digest mismatch on retry).
Side-passthrough requests Untouched — explicitly excluded by the guard.
Non-nudge turns, unset/false config Byte-for-byte unchanged — guard requires nudgeLowEffort === true AND injectedNudge; default parse returns undefined.
session.metadata persistence format No schema change — ad-hoc key on existing Record<string, unknown> (persisted whole like other keys); worst case after a mid-episode restart is one skipped clamp.
Kernel pin / wire shape / tool args Untouched — no package.json change, no message-prefix mutation (effort is a request param), tool-call arguments never pass through this code.

No unintended regressions found.

③ Code review

  • applyNudgeEffortClamp (src/output-steering.ts): JSON.parse guarded; bad JSON / null protocol / array body all pass through verbatim. Delegates to the existing lowerEffort — clamp-only, never raises, never injects absent fields, Gemini -1 dynamic budget untouched. Reusing rather than duplicating the floors is the right call (single source of truth for the wire floors).
  • nudgeLowEffortDecision: clean edge detector (enabled && injected && !prevInjected), nextPrev refreshed every qualifying pass — defer-and-retry cycles don't re-clamp, new episodes do. Exactly the "first response of the episode" semantics from floor 45.
  • forward() insertion: guard shape mirrors the egress-stamp guard; reads global-level opts.compress.nudgeLowEffort raw — matches the documented v1 global-only design (provider/model cascade deliberately out of scope). [nudge-low-effort] log line on every clamp gives the A/B observability promised in the trade-off analysis.
  • Two benign footnotes (no action needed): (a) if the first nudge turn fails upstream and the client retries identical history, the retry isn't clamped (prev already consumed) — acceptable one-shot semantics; (b) the flag is set even when renderNudgeText happens to return empty text or throw (caught) — clamp then fires on an unchanged payload, harmless since lower-only.

④ Test validity

tests/nudge-low-effort.test.ts (24 tests) has teeth throughout: exact-value assertions per wire (16000→1024, 8000→128, high→low), at-floor byte-identity, below-floor never-raised, absent-field never-injected, idempotency, bad-input safety, classifier-independence (the semantic difference from effortRouting is pinned), full episode cycle [T,T,F,T]→[clamp T,F,F,T], config round-trip + non-boolean rejection + default-off. One gap worth noting for follow-up (non-blocking): everything is unit-level — there's no integration test driving the actual forward() wiring (guard + metadata update), so if that block were later moved or its guard weakened, these tests would stay green. A fake-upstream integration case would pin it.

⑤ Diff cleanliness & hot-file coordination

Diff is tight: 6 files, every line on-purpose, no whitespace churn, docs en/zh fully mirrored (verified the zh section line-by-line). Coordination note: four open PRs also touch src/server.ts — most notably #1449 (extract prepare*/inject* pipelines into src/server/, which restructures exactly the regions this PR adds injectedNudge to) and #1491 (nudge-domain, also touches src/config.ts). Whichever merges second will need a semantic (not textual) rebase over the other. No conflict blocks merging this one now.

Behavior change disclosure (per policy)

  • Unset/false (default): zero change, byte-for-byte — verified by the guard logic and the 19-skip/0-fail suite parity.
  • When true: one-shot reduction of thinking/effort on the first compression-nudge turn only. Old → new: that single turn's effort goes to the wire floor (Anthropic 1024 / Gemini 128 / OpenAI+Responses low), next request restores. Intended cost lever; known trade-off (may reduce the model's ~10% defer judgment) is mitigated by the per-clamp log line for A/B.

Evidence

  • Local (worktree @ head 49ca7577): npm run typecheck clean; targeted nudge-low-effort + output-steering 58/58; full suite 3424 tests / 3405 pass / 0 fail / 19 skipped — matches the PR body claim exactly.
  • CI on head: 13/13 green (test ubuntu 22/24, windows 22/24, all four client e2e lanes, hermetic registry e2e, version-guard, build-artifact).
  • Base: single commit on current master tip, zero staleness.

中文摘要:审查了 nudge-low-effort(压缩 nudge 首轮一次性压低思考参数、默认关闭的可选功能)——方向与 #1640 拍板一致,既有行为(nudge 注入判定、双压缩模式、输出转向、egress 摘要、side-passthrough、默认路径)逐项清点均无回归,测试有牙但缺一个 forward() 集成级用例(不阻塞),CI 13/13 全绿,可以合并;注意与 #1449/#1491 的热文件排序问题。

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant