Skip to content
Merged
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ Versions follow the merge of a `*_release-v*` branch; CI publishes to npm on tag

### Features

- **Native Google (Gemini) wire support: `generateContent` / `streamGenerateContent` conversations are tracked, folded and compressed (#866, PR #867)**: bili terminated TLS for `generativelanguage.googleapis.com` and forwarded POSTs verbatim, because protocol detection matched only `/chat/completions`, `/v1/messages` and `/responses` — Gemini's `/v1beta/models/<model>:streamGenerateContent` matched nothing, so `protocol` stayed `null`, the kernel path was skipped, the request logged as `[unknown] forward POST →`, and no conversation identity was minted: ACP compression never fired while the plugin cancels the client's own compaction, and a live omp session reached 339k tokens. The kernel now carries a full Gemini codec (`src/wire/google.ts`: `googleToCore` / `coreToGoogle` / `googleSystemText` / `conversationSignalGoogle`, flat `{name,description,parameters}` function declarations, `thoughtSignature` and `thought` sidecars, and same-side run merging because Gemini rejects non-alternating roles); the proxy detects the wire from the URL (`googlePathKind`, `googleModelFromPath`), hoists system text out of the fold space into `systemInstruction`, resolves identity by prefix affinity over `contents` (Gemini clients send no session header, so the header chain cannot apply), maps usage as input = `promptTokenCount` − `cachedContentTokenCount` and output = `candidatesTokenCount` + `thoughtsTokenCount`, treats `generationConfig.maxOutputTokens` as the output budget, replays thinking signatures on re-issued reasoning, and rejects the OpenAI-compatibility shim on purpose: it drops `thoughtSignature` / `extra_content`, which Gemini 3 function calling requires. Two defects that only a live wire exposed were fixed on the same branch: preflight summaries posted to the client's own SSE endpoint (`:streamGenerateContent` answers SSE whatever the request says) were parsed only when `preflightStreamSummary` was set, so every summary read as empty (`carried N SSE event(s) but no summary text`, 8 attempts, then a 502 and a 300s dead-end cooldown) — the response shape now decides the parser (`SSE_DATA_LINE_RE`, `extractSummaryFromSse` vs `extractSummaryText`); and the proxy's synthesized tool result carried only `toolCallId`, so the wire emitted `functionResponse{name:"unknown"}` answering `functionCall{name:"compress"}` and Google rejected the whole request with 400 `INVALID_ARGUMENT` (Gemini pairs calls by name, OpenAI/Anthropic by id). Covered by `tests/wire-google.test.ts`, `tests/adapter-google.test.ts`, `tests/preflight-google-summary.test.ts` and google cases in `tests/preflight-sse-extraction.test.ts`, with `tests/e2e-google-compress.test.ts` carrying a strict Gemini shape validator (alternating roles, one payload per part, signatures only on model parts, `functionResponse` named AND answering a call seen earlier) that reports nine `functionResponse name=unknown answers no call` violations when the name fix is reverted; verified live against `gemini-3.8-flash`, where a proxy-owned `compress` call executed, folded ~97k tokens and was accepted.
- **Image billing modes (`imageBilling` / `BILI_IMAGE_BILLING`): pixel-tile upstreams no longer get permanently 502'd by byte-based image estimates (#767)**: follow-up to #496/#500 — on official Codex/OpenAI/Anthropic upstreams (pixel-tile billing), once a session's historical `lastInputTokens` baseline sits above the window, the default `base64 length / 4` image cost closes BOTH safe-forward paths at once (#496 forward-once requires a sub-window baseline; #300 stale-baseline fit requires `payloadEstimate < limit`), so preflight fails 502 `preflight_compress_failed` ("Images alone account for ~N tokens") forever even though the images bill only a few thousand tokens upstream (repro: two screenshots, byte estimate ~300K vs real billing ~4.4K). New global + per-provider `imageBilling` (`"auto" | "pixels" | "bytes"`; env `BILI_IMAGE_BILLING` overrides both, live-read per request): `pixels` parses the image header (PNG/JPEG/WebP/GIF/BMP, bounded decode — never the full body) and charges first-party high-detail tile billing (512px tiles, short side scaled up to 768px, long side capped at 2048px → 765–2805 tokens per image; unparsable formats fall back to a flat 16384); `bytes` keeps the conservative default so byte-counting relays retain their #488/#496 protection unchanged. `auto` classifies by upstream host (`openai.com` / `openai.azure.com` / `chatgpt.com` / `api.anthropic.com` suffixes → pixels; everything else → bytes). Remote URLs stay flat 4096 in both modes; `BILI_IMAGE_TOKEN_CAP` still clamps on top of both. A stale baseline now self-heals within one turn of pixel-billed traffic (usage reports replace the baseline absolutely), while #496's one-rejection-then-block guarantee for genuine byte-relay overflow is preserved — pinned by regression tests: recovery e2e (stale 696789 baseline + two large screenshots forwards and recovers under `pixels`), bytes-mode control (same payload still fails fast with zero images forwarded), learned-limit-only variant, and env-precedence check.
- **Session-file encryption at rest behind `BILI_ENCRYPTION_KEY` (#708, PR #711)**: for deployments on untrusted nodes, persisted session files (block summaries plus up to ~16k tokens of folded conversation per session — effectively full conversation content) can now be stored as `BILIENC1` AES-256-GCM over zstd-compressed JSON (zstd via built-in node:zlib on Node ≥ 22.15, raw mode byte on older runtimes; compression also shrinks files ~5–10×). The key is env-only — deliberately NO key-file support, since a key file would sit on the same untrusted filesystem as the data — exactly 32 bytes hex or base64, never written to disk or logs; an invalid value aborts startup (fail fast instead of silently running unencrypted). When enabled, boot takes over legacy plaintext files once: each is re-encoded and atomically replaced in place (the rename IS the old-file deletion; a crash mid-migration self-heals on the next boot), covering `.fb.json` spill files too; unreadable files are left in place with a warning. Wrong key at boot → GCM auth failure → affected sessions skip as corrupt (logged, no crash). Mechanism is the kernel's optional `StateStore` codec hook (acp-kernel#249 → 0.0.64); unset key = zero behavior change.
- **Per-route `passthrough` option for providers (#661, PR #662)**: `"providers": { "mitm://zcode.z.ai": { "passthrough": true } }` forwards matching requests byte-for-byte — no kernel round-trip, no render tags, no `prompt_cache_key` removal, response piped verbatim, no session state — the scoped sibling of the global `passthrough` switch. `mitm://` keys target only MITM (login-client) traffic; plain `https://` host keys cover both MITM and `/bili/` traffic.
Expand Down
20 changes: 18 additions & 2 deletions src/compress-tool.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,12 @@ import {
ABSORB_TOOL_OPENAI,
RULE_TOOL_NAME,
SEARCH_CONTEXT_TOOL,
SEARCH_CONTEXT_TOOL_GOOGLE,
SEARCH_CONTEXT_TOOL_OPENAI,
SEARCH_CONTEXT_TOOL_RESPONSES,
SEARCH_CONTEXT_TOOL_NAME,
ACP_TOOLS_ANTHROPIC,
ACP_TOOLS_GOOGLE,
ACP_TOOLS_OPENAI,
ACP_TOOLS_RESPONSES,
ACP_READONLY_TOOLS_RESPONSES,
Expand All @@ -42,25 +44,31 @@ export {
COMPRESS_TOOL,
COMPRESS_TOOL_OPENAI,
COMPRESS_TOOL_RESPONSES,
COMPRESS_TOOL_GOOGLE,
DECOMPRESS_TOOL,
DECOMPRESS_TOOL_OPENAI,
DECOMPRESS_TOOL_RESPONSES,
DECOMPRESS_TOOL_GOOGLE,
SEARCH_CONTEXT_TOOL,
SEARCH_CONTEXT_TOOL_OPENAI,
SEARCH_CONTEXT_TOOL_RESPONSES,
SEARCH_CONTEXT_TOOL_GOOGLE,
ACP_STATUS_TOOL,
ACP_STATUS_TOOL_OPENAI,
ACP_STATUS_TOOL_RESPONSES,
ACP_STATUS_TOOL_GOOGLE,
ACP_TOOLS_OPENAI,
ACP_TOOLS_ANTHROPIC,
ACP_TOOLS_RESPONSES,
ACP_TOOLS_GOOGLE,
ACP_READONLY_TOOLS_RESPONSES,
buildCompressSystemPrompt,
buildCompressTextSystemPrompt,
buildCompressHybridSystemPrompt,
ABSORB_TOOL_NAME,
ABSORB_TOOL,
ABSORB_TOOL_OPENAI,
ABSORB_TOOL_GOOGLE,
buildAbsorbSystemPrompt,
RULE_TOOL_NAME,
} from "acp-kernel";
Expand Down Expand Up @@ -104,9 +112,16 @@ export const BILI_SEARCH_CONTEXT_TOOL_RESPONSES = {
parameters: withConversationId(SEARCH_CONTEXT_TOOL_RESPONSES.parameters),
};

export const BILI_SEARCH_CONTEXT_TOOL_GOOGLE = {
name: SEARCH_CONTEXT_TOOL_GOOGLE.name,
description: SEARCH_CONTEXT_TOOL_GOOGLE.description,
parameters: withConversationId(SEARCH_CONTEXT_TOOL_GOOGLE.parameters),
};

export const BILI_ACP_TOOLS_ANTHROPIC = ACP_TOOLS_ANTHROPIC.map((t) => (t.name === SEARCH_CONTEXT_TOOL_NAME ? BILI_SEARCH_CONTEXT_TOOL : t));
export const BILI_ACP_TOOLS_OPENAI = ACP_TOOLS_OPENAI.map((t) => (t.function.name === SEARCH_CONTEXT_TOOL_NAME ? BILI_SEARCH_CONTEXT_TOOL_OPENAI : t));
export const BILI_ACP_TOOLS_RESPONSES = ACP_TOOLS_RESPONSES.map((t) => (t.name === SEARCH_CONTEXT_TOOL_NAME ? BILI_SEARCH_CONTEXT_TOOL_RESPONSES : t));
export const BILI_ACP_TOOLS_GOOGLE = ACP_TOOLS_GOOGLE.map((t) => (t.name === SEARCH_CONTEXT_TOOL_NAME ? BILI_SEARCH_CONTEXT_TOOL_GOOGLE : t));
export const BILI_ACP_READONLY_TOOLS_RESPONSES = ACP_READONLY_TOOLS_RESPONSES.map((t) => (t.name === SEARCH_CONTEXT_TOOL_NAME ? BILI_SEARCH_CONTEXT_TOOL_RESPONSES : t));

// The kernel ships no Responses-format absorb const (the four ACP tools have
Expand All @@ -120,8 +135,8 @@ export const ABSORB_TOOL_RESPONSES = {
};

// The reconciled kernel (acp-kernel#332) ships RULE_TOOL_NAME + the rule state
// helpers but no wire tool objects. Synthesize all three shapes here so every
// injection point (wire helpers, plugin manifest) serves one definition.
// helpers but no wire tool objects. Synthesize all four wire shapes here so
// every injection point (wire helpers, plugin manifest) serves one definition.
const RULE_TOOL_DESCRIPTION = "Record a short, principle-level reminder so it survives context compression — the call and its result are protected and stay in context. Record when: the user calls out or repeatedly emphasizes a lesson; the user asks you to remember or follow a behavior; you personally hit a major pitfall worth remembering long-term. Keep each rule to one short line. Omit the rule argument to list recorded rules.";
const RULE_PARAM_SCHEMA = {
type: "object",
Expand All @@ -132,6 +147,7 @@ const RULE_PARAM_SCHEMA = {
export const RULE_TOOL = { name: RULE_TOOL_NAME, description: RULE_TOOL_DESCRIPTION, input_schema: RULE_PARAM_SCHEMA };
export const RULE_TOOL_OPENAI = { type: "function" as const, function: { name: RULE_TOOL_NAME, description: RULE_TOOL_DESCRIPTION, parameters: RULE_PARAM_SCHEMA } };
export const RULE_TOOL_RESPONSES = { type: "function" as const, name: RULE_TOOL_NAME, description: RULE_TOOL_DESCRIPTION, parameters: RULE_PARAM_SCHEMA };
export const RULE_TOOL_GOOGLE = { name: RULE_TOOL_NAME, description: RULE_TOOL_DESCRIPTION, parameters: RULE_PARAM_SCHEMA };

export function parseCompressInput(input: unknown, callId?: string) {
const parsed = parseCompressArgs(input, { callId });
Expand Down
1 change: 1 addition & 0 deletions src/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -256,6 +256,7 @@ const CONTEXT_LIMIT_TABLE: Array<{ match: RegExp; limit: number }> = [
{ match: /^gpt-4o/i, limit: 128_000 },
{ match: /^gpt-4-turbo/i, limit: 128_000 },
{ match: /^o[13]-/i, limit: 200_000 },
{ match: /^gemini-3/i, limit: 1_048_576 },
{ match: /^gemini-2\.5/i, limit: 1_000_000 },
{ match: /^gemini-1\.5/i, limit: 1_000_000 },
{ match: /^glm-4\.6/i, limit: 128_000 },
Expand Down
5 changes: 5 additions & 0 deletions src/fake-completion.ts
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,9 @@ export function fakeBufCap(): number {
const ANTHROPIC_TOOL_BLOCK = /"type"\s*:\s*"tool_use"/;
const OPENAI_TOOL_BLOCK = /"tool_calls"\s*:\s*\[\s*\{/;
const RESPONSES_TOOL_BLOCK = /"type"\s*:\s*"function_call"/;
// Gemini has no `type` discriminator: a real invocation is a functionCall PART
// (`"functionCall":{"name":…}`) inside candidates[*].content.parts.
const GOOGLE_TOOL_BLOCK = /"functionCall"\s*:\s*\{/;

export function hasToolBlock(protocol: WireProtocol, rawText: string): boolean {
switch (protocol) {
Expand All @@ -49,6 +52,8 @@ export function hasToolBlock(protocol: WireProtocol, rawText: string): boolean {
return OPENAI_TOOL_BLOCK.test(rawText);
case "responses":
return RESPONSES_TOOL_BLOCK.test(rawText);
case "google":
return GOOGLE_TOOL_BLOCK.test(rawText);
}
}

Expand Down
25 changes: 22 additions & 3 deletions src/image-tokens.ts
Original file line number Diff line number Diff line change
Expand Up @@ -208,9 +208,27 @@ function urlOf(v: unknown): string | undefined {
return undefined;
}

export function imageTokensInParsedBody(protocol: "anthropic" | "openai" | "responses", body: unknown, billing: ResolvedImageBilling = "bytes"): number {
export function imageTokensInParsedBody(protocol: "anthropic" | "openai" | "responses" | "google", body: unknown, billing: ResolvedImageBilling = "bytes"): number {
if (!isObj(body)) return 0;
let total = 0;
if (protocol === "google") {
// Gemini carries images as `inlineData` (raw base64 in `data`) or
// `fileData` (a remote `fileUri`) PARTS of `contents[].parts` — there is
// no separate content-array layer and no `type` discriminator.
const contents = body.contents;
if (!Array.isArray(contents)) return 0;
for (const c of contents) {
if (!isObj(c) || !Array.isArray(c.parts)) continue;
for (const part of c.parts) {
if (!isObj(part)) continue;
const inline = part.inlineData;
if (isObj(inline) && typeof inline.data === "string") total += base64ImageCost(inline.data, billing);
const file = part.fileData;
if (isObj(file) && typeof file.fileUri === "string") total += costForUrl(file.fileUri, billing);
}
}
return total;
}
if (protocol === "responses") {
const input = body.input;
if (!Array.isArray(input)) return 0;
Expand Down Expand Up @@ -250,10 +268,11 @@ export function imageTokensInParsedBody(protocol: "anthropic" | "openai" | "resp
// Cheap gate: most bodies carry no images — skip the JSON parse entirely then.
// prepared.body is bili's own compact JSON.stringify, but client raw buffers
// may carry spaces, so probe both forms.
export function imageTokensInRawBody(protocol: "anthropic" | "openai" | "responses", raw: string | Buffer, billing: ResolvedImageBilling = "bytes"): number {
export function imageTokensInRawBody(protocol: "anthropic" | "openai" | "responses" | "google", raw: string | Buffer, billing: ResolvedImageBilling = "bytes"): number {
const s = typeof raw === "string" ? raw : raw.toString("utf8");
const probe =
protocol === "responses" ? s.includes("input_image")
protocol === "google" ? s.includes("inlineData") || s.includes("fileData")
: protocol === "responses" ? s.includes("input_image")
: protocol === "openai" ? s.includes("image_url")
: s.includes('"type":"image"') || s.includes('"type": "image"');
if (!probe) return 0;
Expand Down
Loading
Loading