Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CONFIGURATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -1110,6 +1110,7 @@ Targets are **references into the `providers` table**: each entry is the string
"enabled": true,
"targets": ["glm/glm-4.9-flash", "claude/claude-haiku-4.5"],
"budget": { "totalTimeoutMs": 50000, "targetTimeoutMs": 25000, "maxSummaryBytes": 65536 },
"concurrency": 8,
"autoFold": true,
"autoFoldTargetTokens": 40000
}
Expand All @@ -1121,6 +1122,10 @@ A recipe carries: `baseUrl` (HTTPS; loopback HTTP allowed for local development;

Up to 16 targets per chain are supported. Saving an enabled chain through the Web API validates that every reference resolves (unknown provider or model → HTTP 400). At runtime an unresolvable reference is logged once and disables the chain until fixed — it never falls back to summarizing with the main model. `secret:` values are stored separately from the main JSON configuration in the private `billion-context.json.summary-credentials.json` file and are never returned by the configuration API. On Windows, protect this file and its parent directory with an administrator-only ACL; stale `.lock` files require manual removal after confirming no proxy process is writing the store. The total budget is shared across all targets and compression entry points, and cancellation or session-state changes discard generated results without folding. The agent-side plugin bridge (dsh / pi / opencode / codex MCP) sizes each `compress` call's HTTP wait to `totalTimeoutMs` plus a small overhead margin — learned from the proxy at bootstrap — so raising this budget no longer caps long folds at a hidden 60-second client-side timeout (#2652).

`concurrency` (integer `1..32`, default `4`) sizes the **process-wide summary queue**: every external-summary call on this host — preflight auto-folds for *all* sessions plus every model-initiated `compress` call — shares one pool of in-flight requests. Raise it when your summarizer has headroom (local relay, self-hosted model, high-quota flash endpoint); keep the conservative default on a small provider quota. A chain's value resizes the shared pool only while that chain's work is in flight; if several chains with different values run concurrently, the pool grows to the largest demand and returns to the base size once those batches settle — one session can never permanently resize the pool for everyone else. Note the knob is **grow-only**: one session raising the pool must not throttle another's in-flight work, so values below the default (`1..3`) are accepted for validation but never shrink the pool below `4` — do not set a low value expecting quota protection; configure the chain's own `budget`/`targetTimeoutMs` instead.

Interaction with the budget: the achievable fold rate of a batch is bounded by how many calls fit in the pool (`concurrency`) and how long each may take (`targetTimeoutMs`) — the budget alone does not control the outcome. A saturated pool does not surface as a visible error: a batch still queued when its total deadline expires is discarded as `batch=deadline` with no attempts recorded (`attempts=[]`), and the fold pass stops there instead of queueing further rounds behind the congestion. For auto-fold this is treated as transient scheduler congestion, **not** a chain failure: no 10-minute backoff arms, and the next turn retries into the (hopefully drained) pool. If you see those lines while direct calls to your target succeed quickly, the pool — not the timeout — is the binding constraint.

Two optional knobs turn the chain into a **proxy-driven auto-folder** (`autoFold`): with `"autoFold": true` the proxy stops waiting for the model to call the compression tool at all — the same zero-model folding engine that guards the overflow window also folds the conversation down to `autoFoldTargetTokens` (default: half the model window) *before* forwarding, using the external chain and kernel-chosen ranges. The model never sees a nudge and never spends thinking tokens drafting compression plans. The knob rides whole-chain replace semantics like the rest of the block; it only takes effect while the chain itself is enabled — setting either knob on a disabled (or omitted-`enabled`) chain has no effect and is logged once as a warning rather than failing silently. `autoFoldTargetTokens` must be an integer in `[8192, 10000000]`. Failure is fail-open: if the chain errors at fold time, the request forwards as-is, a 10-minute backoff suppresses auto-folding, and classic advisory nudges return for that window. One exception (#2662): a zero-progress fold caused purely by queue drops — batches submitted but no summary call ever dispatched because the shared pool is saturated — does not arm the backoff, since that is scheduler congestion rather than a chain failure; auto-fold simply retries on the next turn.

#### Agent-reported providers (fallback layer)
Expand Down
5 changes: 5 additions & 0 deletions CONFIGURATION.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -1113,6 +1113,7 @@ URL 键通道(或带 `bind` 的命名条目)可以向上游发送**自己的
"enabled": true,
"targets": ["glm/glm-4.9-flash", "claude/claude-haiku-4.5"],
"budget": { "totalTimeoutMs": 50000, "targetTimeoutMs": 25000, "maxSummaryBytes": 65536 },
"concurrency": 8,
"autoFold": true,
"autoFoldTargetTokens": 40000
}
Expand All @@ -1124,6 +1125,10 @@ URL 键通道(或带 `bind` 的命名条目)可以向上游发送**自己的

每条链最多 16 个目标。通过 Web API 保存启用的链时会校验每个引用可解析(未知 provider 或 model → HTTP 400)。运行时遇到不可解析的引用会记录一次警告并禁用整条链直至修复 —— 绝不会回退用主模型写摘要。`secret:` 值单独存储在私有的 `billion-context.json.summary-credentials.json` 文件中,不随主 JSON 配置下发,配置 API 也不会返回。Windows 上请用仅管理员的 ACL 保护该文件及其父目录;确认没有代理进程在写存储后,残留的 `.lock` 文件需手工删除。总预算由全部目标与压缩入口共享;取消或会话状态变化会丢弃已生成但未落盘的结果。代理端插件桥(dsh / pi / opencode / codex MCP)会把每次 `compress` 调用的 HTTP 等待时间设为 `totalTimeoutMs` 加少量开销余量 —— 启动时从代理学习得到 —— 因此调高该预算不会再让长折叠被隐藏的 60 秒客户端超时截断(#2652)。

`concurrency`(整数 `1..32`,默认 `4`)决定**进程级摘要队列**的大小:本机所有外部摘要调用 —— *全部* 会话的 preflight 自动折叠加上每次模型发起的 `compress` 调用 —— 共享同一个在途请求池。当你的摘要服务有余量(本地中继、自托管模型、高配额 flash 端点)时调大它;小配额 provider 保持保守默认即可。某条链设置的值只在该链的工作在途期间调整共享池大小;多条不同取值的链并发运行时,池子增长到最大需求,待这些批次结束后回到基础值 —— 任何单个会话都无法永久改变其他会话看到的池大小。注意该旋钮是**只增不减**的:单个会话调大池子不应压制其他会话的在途工作,因此低于默认值的取值(`1..3`)能通过校验但永远不会把池子缩到 `4` 以下 —— 不要指望设小值来保护配额,请改配该链自己的 `budget`/`targetTimeoutMs`。

与 budget 的交互:一个批次能达到的折叠速率由池能容纳的并发数(`concurrency`)和每次调用的时限(`targetTimeoutMs`)共同决定 —— 单看 budget 并不能解释结果。池饱和不会表现为可见错误:批次还在排队就耗尽总时限的话,会被整体丢弃为 `batch=deadline` 且没有任何尝试记录(`attempts=[]`),折叠过程也会就地停止而不是继续排在拥塞后面。对 auto-fold 而言这被当作暂时的调度拥塞而**非**链路故障:不会触发下文所述的 10 分钟退避,下一个回合会在(有望已排空的)池子上重试。如果你看到这些日志行、而直接调用目标却很快成功,说明瓶颈是池而不是超时。

两个可选旋钮可把这条链升级为**代理驱动的自动折叠器**(`autoFold`):`"autoFold": true` 后代理不再等模型调压缩工具 —— 原本只守溢出窗口的零模型折叠引擎同样会在转发前把对话折到 `autoFoldTargetTokens`(默认:模型窗口的一半)以下,范围由内核选定、摘要由外部链完成。主模型完全看不到 nudge,也不会把思考预算花在起草压缩方案上。该旋钮随整链替换语义生效,且只在链本身启用时才起作用——在链未启用(或省略 `enabled`)时设置任一旋钮均无效,并会记录一次警告而非静默失效;`autoFoldTargetTokens` 必须是 `[8192, 10000000]` 内的整数。失败时 fail-open:链在折叠时出错则照常转发请求,退避 10 分钟内暂停自动折叠并恢复经典 advisory nudge。一个例外(#2662):若零进展完全由队列丢弃造成——批次已提交但因共享池饱和、没有任何一次摘要调用真正派发——则不触发退避(那是调度拥塞而非链故障),auto-fold 在下一轮直接重试。

#### Agent 上报的 providers(兜底层)
Expand Down
2 changes: 1 addition & 1 deletion src/external-summary-compress.ts
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ export async function applyConfiguredCompression(input: unknown, ctx: RewriteCtx
// The plan rides the request Config rail (#833: ctx.config is the
// session's effective Config — same cascade that produced the wire).
let plan;
try { plan = configuredSummaryPlan((ctx.config as ResolvedKernelConfig).externalSummary); }
try { plan = configuredSummaryPlan((ctx.config as ResolvedKernelConfig).externalSummary, ctx.session.id); }
catch { return compressResult("[Compression FAILED: external summary configuration is invalid. Nothing compressed.]", "refused", 0); }
if (!plan) return applyRanges(parseCompressInput(input, callId), ctx);
const parsed = parseCompressInput(withOptionalSummaries(input), callId);
Expand Down
29 changes: 27 additions & 2 deletions src/external-summary-http.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import { fetchWithTimeout } from "./fetch-util.js";
import { extractSummaryFromSse, extractSummaryText, summaryPayload, type PreflightProtocol } from "./preflight.js";
import { normalizeSseLineEndings } from "./sse-util.js";
import { proxyDispatcher } from "./upstream-proxy.js";
import { PACKAGE_NAME, VERSION } from "./version.js";

/** Internal resolved request plan, not a user-config schema or credential store. */
export interface SummaryHttpTarget {
Expand All @@ -15,6 +16,8 @@ export interface SummaryHttpTarget {
readonly contextWindow?: number;
readonly outputTokens?: number;
readonly proxyUrl?: string;
/** #2760: stable conversation id, stamped as x-opencode-session on OpenCode Go targets only. */
readonly sessionId?: string;
}

function record(value: unknown): Record<string, unknown> | undefined {
Expand Down Expand Up @@ -108,6 +111,19 @@ function parseSummary(protocol: PreflightProtocol, text: string): string {
return summary;
}

/** #2760: OpenCode Go (`opencode.ai/zen/go`) requires a stable x-opencode-session
* header for routing + prompt cache and answers HTTP 400 MissingSessionID without
* one. Matched on host + exact `/zen/go` or `/zen/go/...` so adjacent endpoints
* (a hypothetical `/zen/golang`) are left untouched too. */
export function isOpenCodeGoTarget(rawUrl: string): boolean {
try {
const parsed = new URL(rawUrl);
return parsed.hostname.toLowerCase() === "opencode.ai" && (parsed.pathname === "/zen/go" || parsed.pathname.startsWith("/zen/go/"));
} catch {
return false;
}
}

export function createSummaryHttpCandidate(target: SummaryHttpTarget, maxResponseBytes: number): SummaryCandidate {
let url: URL;
try { url = new URL(target.url); } catch { throw new Error("Invalid external summary endpoint"); }
Expand All @@ -121,6 +137,13 @@ export function createSummaryHttpCandidate(target: SummaryHttpTarget, maxRespons
const plan = { ...target, url: url.href, headers };
plan.headers.set("content-type", "application/json");
if (plan.protocol === "anthropic" && !plan.headers.has("anthropic-version")) plan.headers.set("anthropic-version", "2023-06-01");
// #2760: OpenCode Go rejects requests lacking a stable x-opencode-session
// (HTTP 400 MissingSessionID). Stamp it plus bili's own user agent for Go
// targets only — never the client's inbound session/auth headers.
if (isOpenCodeGoTarget(plan.url) && typeof plan.sessionId === "string" && plan.sessionId.length > 0) {
plan.headers.set("x-opencode-session", plan.sessionId);
plan.headers.set("user-agent", `${PACKAGE_NAME}/${VERSION}`);
}
return {
async summarize(work: Readonly<SummaryWork>, signal: AbortSignal): Promise<string> {
signal.throwIfAborted();
Expand All @@ -136,8 +159,10 @@ export function createSummaryHttpCandidate(target: SummaryHttpTarget, maxRespons
payload[plan.protocol === "responses" ? "max_output_tokens" : "max_tokens"] = plan.outputTokens;
}
}
// Reuse the existing codecs without carrying the main request's auth,
// session headers, or model. Each candidate dispatches exactly once.
// Reuse the existing codecs without carrying the main request's auth or
// model. Each candidate dispatches exactly once. (The Go-specific
// x-opencode-session / user-agent above are derived from bili's own
// session id, not copied from the client's inbound headers — #2760.)
const { response, clearTimer } = await fetchWithTimeout(plan.url, {
method: "POST", headers: plan.headers, body: JSON.stringify(payload), redirect: "manual", dispatcher: proxyDispatcher(plan.proxyUrl),
}, undefined, signal);
Expand Down
36 changes: 17 additions & 19 deletions src/external-summary-runtime.ts
Original file line number Diff line number Diff line change
@@ -1,12 +1,13 @@
import { performance } from "node:perf_hooks";
import { SummaryCredentialStore } from "./external-summary-credentials.js";
import { createSummaryHttpCandidate } from "./external-summary-http.js";
import { parseExternalSummarySettings, type ExternalSummarySettings } from "./external-summary-settings.js";
import { parseExternalSummarySettings, SUMMARY_CONCURRENCY_DEFAULT, type ExternalSummarySettings } from "./external-summary-settings.js";
import { ExternalSummaryExecutor, type ExternalSummaryBatchResult, type SummaryCandidate, type SummaryWork } from "./external-summary.js";
import { log as loggerLog } from "./logger.js";

// One shared queue across all sessions and all compression entry points.
const executor = new ExternalSummaryExecutor(4);
// One shared queue across all sessions and all compression entry points; its
// base size is the config default — a chain's `concurrency` resizes it for
// that batch's lifetime (#2657).
const executor = new ExternalSummaryExecutor(SUMMARY_CONCURRENCY_DEFAULT);

// Test seam (#2662): saturate the process-wide pool deterministically.
export const _executorForTest = executor;
Expand All @@ -18,31 +19,31 @@ const seenCandidateWarnings = new Set<string>();
export class ConfiguredSummaryPlan {
private readonly candidates: readonly SummaryCandidate[];
private readonly settings: ExternalSummarySettings;
readonly deadline: number;
/** #2662: summarize() invocations vs candidate calls actually dispatched —
* calls > 0 && attempts === 0 means no candidate call was ever dispatched
* (shared-pool congestion, or the plan's budget already exhausted before
* reaching the pool — congestion, not a chain failure). */
readonly stats = { calls: 0, attempts: 0 };
private readonly sessionId: string | undefined;

// `raw` may be an already-parsed chain off the request rail or raw JSON
// from a hand-edited file — re-parse here so invalid settings fail
// loudly at plan build, never silently use main-model summaries.
constructor(raw: unknown, store = new SummaryCredentialStore(), env: NodeJS.ProcessEnv = process.env) {
constructor(raw: unknown, store = new SummaryCredentialStore(), env: NodeJS.ProcessEnv = process.env, sessionId?: string) {
// inlineKeys: the rail may carry agent-registry recipes whose keys
// were resolved in the agent's memory (#2336) — literal apiKey bytes
// instead of a credentialRef.
this.settings = parseExternalSummarySettings(raw, { inlineKeys: true });
this.sessionId = sessionId;
const proxyUrl = env.BILI_UPSTREAM_PROXY?.trim() || undefined;
this.deadline = performance.now() + this.settings.budget.totalTimeoutMs;
this.candidates = this.settings.targets.map((target) => {
try {
const key = target.apiKey !== undefined ? target.apiKey : store.resolve(target.credentialRef ?? "", env);
if (!key) throw new Error(`credential ${target.credentialRef || "(none)"} did not resolve to a value`);
const headers: Record<string, string> = target.protocol === "anthropic" ? { "x-api-key": key }
: target.protocol === "google" ? { "x-goog-api-key": key }
: { authorization: `Bearer ${key}` };
return createSummaryHttpCandidate({ ...target, headers, proxyUrl }, this.settings.budget.maxSummaryBytes * 4 + 65536);
return createSummaryHttpCandidate({ ...target, headers, proxyUrl, ...(this.sessionId !== undefined ? { sessionId: this.sessionId } : {}) }, this.settings.budget.maxSummaryBytes * 4 + 65536);
} catch (error) {
// Log WHY this target is unusable (once per signature) so a
// misconfigured chain is diagnosable instead of silently degrading
Expand All @@ -60,15 +61,12 @@ export class ConfiguredSummaryPlan {
}

async summarize(work: readonly SummaryWork[], signal?: AbortSignal): Promise<ExternalSummaryBatchResult> {
const remaining = Math.floor(this.deadline - performance.now());
if (remaining <= 0) {
this.stats.calls += 1;
return { status: "deadline", results: [] };
}
const batch = await executor.executeBatch(work, this.candidates, {
...this.settings.budget, totalTimeoutMs: remaining,
targetTimeoutMs: Math.min(remaining, this.settings.budget.targetTimeoutMs),
}, signal);
// Deliberately NO cached deadline here: preflight reuses one plan across
// many summarize() calls, so each must get the FULL configured budget —
// a construction-time clock point went stale mid-round and starved every
// later range (#2639). The model-tool path packs all ranges into one call,
// where executeBatch's shared deadline still bounds the whole batch.
const batch = await executor.executeBatch(work, this.candidates, { ...this.settings.budget }, signal, { concurrency: this.settings.concurrency });
this.stats.calls += 1;
for (const item of batch.results) this.stats.attempts += item.attempts.length;
return batch;
Expand All @@ -80,8 +78,8 @@ export class ConfiguredSummaryPlan {
* `mergeCompress` with whole-chain-replace semantics). undefined/disabled →
* undefined (legacy in-model summaries). Invalid settings throw — they must
* fail loudly, never silently fall back to main-model summaries. */
export function configuredSummaryPlan(settings: unknown): ConfiguredSummaryPlan | undefined {
export function configuredSummaryPlan(settings: unknown, sessionId?: string): ConfiguredSummaryPlan | undefined {
if (!settings || typeof settings !== "object") return undefined;
if ((settings as { enabled?: unknown }).enabled !== true) return undefined;
return new ConfiguredSummaryPlan(settings);
return new ConfiguredSummaryPlan(settings, new SummaryCredentialStore(), process.env, sessionId);
}
Loading
Loading