Skip to content

[PR] fix: surface unsigned functionCall/thought parts on the Google lane before they 400 upstream - #5277

Open
ranxianglei wants to merge 4 commits into
masterfrom
2026-10-31_google-sig-audit-warn
Open

ranxianglei wants to merge 4 commits into
masterfrom
2026-10-31_google-sig-audit-warn

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

What

Google lane (Gemini native / Vertex): after one normal fold, every request of the session 400s upstream with Function call is missing a thought_signature in functionCall parts until the lane is removed from bili (#5253).

Triage established in the issue thread: bili's fold rebuild preserves thoughtSignature verbatim both directions (code audit of googleToCore/coreToGoogle/stripRawStage, plus a three-scenario local repro over real kernel+wire). An unsigned part on the outbound was therefore unsigned on the inbound — the loss happens when the CLIENT reworks its history around the fold splice. But bili produced zero trace of that: the lane just died silently every turn.

This PR makes the failure mode visible and self-attributing:

  • new src/server/google-sig-audit.ts: pure audit of signed/unsigned functionCall/thought parts over a Google contents array (fail-open on malformed shapes) + warn decision (sticky per-session "model signs its calls" latch; warn on the first affected turn, re-warn every 10th);
  • prepareGoogle runs the audit once at egress over BOTH sides of the rebuild. First affected turn logs counts, content positions and tool names, plus the origin — client-resend loss (host/plugin signature replay after history rework) vs bili-side drop (the client resent them SIGNED but bili's rebuild dropped the signature — bili-side defect, re-report with ACP_DUMP_BODY captures). That second branch is the tripwire for the currently-unproven bili path.

Behavior change

None on the wire: additive WARN log only. No outbound bytes, defaults, timing, cache semantics or config-surface change.

Tests

tests/issue5253-google-sig-audit.test.ts — 7 tests, both directions per gate discipline: mixed counts + site strings, all-signed clean, malformed-shape fail-open, site cap, first-turn warn + origin attribution (both branches), silence for non-signing models and clean outbounds, 10-turn re-warn cadence. Targeted run green locally; full-suite verdict from CI.

Part of #5253 — the 400 itself is attributed upstream of bili pending the reporter's inbound/outbound dumps (requested in the issue thread); this PR ships the bili-side detection so the next occurrence names its origin in the log instead of dying silently.

@ranxianglei ranxianglei changed the title fix: surface unsigned functionCall/thought parts on the Google lane before they 400 upstream [PR] fix: surface unsigned functionCall/thought parts on the Google lane before they 400 upstream Oct 30, 2026
@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-31_google-sig-audit-warn (52f10ab)

Latest build — install this exact version

npm install -g billion-context@0.0.9673-pr.5277.2cbd2c3

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
52f10ab billion-context@0.0.9673-pr.5277.2cbd2c3 build
76adec3 billion-context@0.0.7317-pr.5277.9037fea build
3428e5d billion-context@0.0.7295-pr.5277.80ef591 build
950abb1 billion-context@0.0.7248-pr.5277.19f9a75 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr5277.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei ranxianglei added bug Something isn't working dsh p1 scope/wiring labels Oct 31, 2026
ci-check added 2 commits October 31, 2026 10:15
…efore they 400 upstream (#5253)

Gemini thinking models reject a whole request when one replayed functionCall/thought part lacks its thoughtSignature. bili forwards client parts verbatim (proven by audit + repro in #5253), so an unsigned part on the outbound was unsigned on the inbound — but until now the lane just died with an upstream 400 every turn and no bili-side trace.

prepareGoogle now audits both sides of the rebuild at egress: a sticky per-session latch records that the model signs its calls, and the first affected turn logs a WARN naming counts, content positions, tool names, and the origin — client-resend loss (host/plugin signature replay) vs a bili-side drop (re-report with ACP_DUMP_BODY captures). Re-warns every tenth affected turn only. Pure observability: no outbound bytes, defaults, timing or config surface change.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重:不重复。 没有其它 open issue/PR 修「eviction 后 acp_status 报零」——#3982/#3983(compress-from-store)是本 PR 复用的既有网(0.1.208 已含),不是重叠修复;#5111 同属 eviction 生命周期家族但症状不同(驱逐频率决策域,本 PR 明确不碰驱逐策略);#3918 已关闭。

标签判定:bug / dsh / scope-wiring / p2 —— 诊断面误报:压缩本身走 store-fold 正常、下一次模型请求自恢复视图(#3126),无数据丢失、无缓存泄露,够不到 p1。(镜像 labels 接口在本环境未实现,此处记录判定。)

审核进行中(中期报告):

接下来本地跑定向测试 + 等 CI 结果后给八问结论。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

审核结论(结构化八问)

1. 是什么问题? 会话被驻留池驱逐(或进程重启)后,消息视图丢失但 refs/blocks 还在;acp_status 的两个入口(/acp 面板 + ACP 工具)只读内存视图,于是报「0 tool / 0 text + max-compressible 0」——和全新会话无法区分——而上游仍在按全量历史计费(事故中 billed/est 差 6.6×)。零值会被读成「没有可压缩的」。

2. 什么原因导致的? 驱逐丢弃会话整个内存视图(transient),cold-load 只重建磁盘态;pluginSnapshot 只为 fork 会话持久化,普通会话驱逐后就没有任何可读视图。compress 车道早有兜底网(#3982 peekStoredTailView 从统一存储读回全量 raw client view),两个状态面没接上——接线遗漏,不是设计缺陷。

3. 影响范围与严重程度? 仅影响「驱逐/重启 → 下一次模型请求」窗口内:面板/工具报零、max-compressible 0,且 #2366 比值注记会把偏差错误归因给 tokenizer/CJK 差异、PRESSURE NOTE 可能基于伪 maxPending=0 触发。压缩本身不受影响(store-fold 可用,下一次模型请求自恢复视图 #3126),无数据丢失、无缓存泄露 → p2(诊断面误报)。

4. 怎么修复的? 三层策略:① 视图为空且保留 refs/blocks 时,先试 peekStoredTailView(原样复用 #3982:peek-only、任何坏行整体返 null)读回全量 raw client view 渲染状态——面板/工具各在入口做,工具侧门控到 tool === "acp_status",其余工具保持原路径;② store 不可读(无 store/legacy/坏行)时显式打 ⚠️ LIVE VIEW UNREADABLE / LIVE MESSAGE VIEW UNAVAILABLE 标记,写明保留的 refs/blocks 数 + #3126 恢复路径;③ 精确抑制该状态下前提不成立的两条注记(#2366 比值注记、PRESSURE NOTE)。marker 判定在 handleAcpStatus 内部自算(空视图 ∧ 有保留态),天然覆盖两个入口。

5. 是否有回归问题? 逐项审过既有行为:

6. 是否破坏缓存? 不涉及——只读状态面,不触前缀缓存、重试策略、增量游标或去重键。

7. 是否改了设计方向? 不涉及——复用既有 #3982 兜底网,把覆盖从 compress 车道扩到状态面;驱逐策略本身未动(频率问题留在 #5111 决策域)。

8. 是否引入新配置? 不涉及——无新 env/config/默认值。

验证记录

测试有效性

断言有牙:marker 措辞精确匹配 + 保留计数点名、recovered-vs-resident 字节恒等(仅归一时间戳)、正则钉住非零 text share(恰好钉死事故里的零值)、每个门都有正负双向。小缺口一处:无「store 行损坏 → unreadable marker」专项测试(peek 的严格语义已由 compress-from-store 套件钉住,组合路径未单测)——可接受,不构成拦截理由。

行为变更清单(PR 正文均已披露,我逐一核对属实)

面 旧 新
tool/panel,evicted + store 可读 零值 + max-compressible 0 从 store 视图出真实 breakdown/ranges/nudge
tool/panel,store 不可读 零值被读成「无可压缩」 显式 ⚠️ marker(保留计数 + #3126 恢复路径)
#2366 比值注记 ≥1.5× 必发 view-unavailable 时抑制(marker 替代解释)
PRESSURE NOTE 可基于伪 maxPending=0 触发 view-unavailable 时抑制

判定

可合(待 CI 全 lane 转绿)。建议合并前先 rebase 到最新 master——模拟合并已证零冲突,但 base 现落后 108 commit。无 kernel/配置面/wire/持久化格式/version 变更,满足 §7.4 自动合并门禁各项;按规范合并动作留给人工。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b) CI 红分账结论(run 41155681612,head 4f95fba):49 项 check = 31 绿 / 13 skip / 5 红,全部是 test 分片时序 flake,与本 PR 零因果。静态门禁(unused-exports / lifecycle-ledger-guard / docs-sync / config-surface / lock-sync / guards / build 等)在本 head 全绿。

逐分片归因(日志已逐条核对):

分片 失败测试 签名 继承证据
ubuntu 22 s1 issue2129-restart-measurement-state.test.ts:327 「PFA stamp must be on disk after restart」——writeNow→flushAll→reload 后 metadata.anonymousPrefixAffinity 丢失 与 PR #4895、#5077(含 ci-fixes@06ad804ac)同签名,那两处均判 CI 负载时序竞态且本地单跑绿;本 PR diff 与其持久化路径零交集
ubuntu 22 s2 + ubuntu 24 s2 + win 22 s1 claude-native.test.ts:1413 hook e2e static URL spawned 子进程 60s 停在 #5096 startup-failure 页(HTTP 503)被 kill 在档启动/生成时序家族(#5273 win22-s1 同签名);今日跨平台出现
ubuntu 22 s2 + ubuntu 24 s2 + win 22 s2 issue4982-draining-twin-ladder.test.ts P2 开 store 时 P1 仍写 index.db → StorageSecondWriterError 拒绝启动 #5273/#5258 在档家族;本地对照实验坐实:pr-5284 与 pre-PR base c0fa5b2 上同样失败(6/7),PR 因果排除

本地验证(pr-5284): issue2129 单跑 4/4 绿(负载相关);issue4982 单跑 6/7,失败签名与 CI 逐字一致,且在无本 PR 两个 commit 的 base 上复现 → 既有测试竞态,非本 PR 引入。本 PR 定向套件(issue4975×8、issue2366、compress-from-store、issue3126、issue3918×2、issue2029、issue1875、issue3204)此前已全部绿。

处置: ①未 merge ci-fixes——本 head 静态门禁自绿,ci-fixes 不载测试时序修复(merge 只会引入无关文档行);②已触发 rerun-failed-jobs(flake 复跑转绿的概率高);③已在 #4599 留一行分账,#2129 PFA 家族至此跨 3 个 PR 复现,建议哨兵评估独立立项(scope/storage)。

八问更新(相对上一轮结论仅第 5 问变化): 1-4、6-8 不变(可合判定维持)。5. 回归问题:无——5 个红分片全部归因为在档继承 flake(逐条日志签名 + 本地对照实验排除 PR 因果);高危模式(语义漂移/fail-fast 变吞错/阈值时序输出格式漂移)逐项过,本 PR 不触碰。终局:CI 转绿即可由 owner 合并(rerun 中);若 rerun 后 #4982/#2129 家族仍红,那是 master 侧既有时序债,不应阻塞本 PR 的合并判断。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b) rerun 终局更新(修正上一条的「偶发 flake」定性):

判定不变:静态门禁全绿、定向套件全绿、回归审查无破坏——PR 本身可合;当前 4 个红片全部是 master 侧既有时序债,不构成本 PR 的合并阻塞。合并时机由你裁量:按「已知继承红不阻塞」直接合,或等哨兵修复落地后自然转绿再合,两者皆可。

ci-check added 2 commits November 14, 2026 04:03
…ting persisted session state (#5253)

The warn-cadence counter (session.metadata.googleSigWarnedTurns) was
incremented on every turn with an unsigned functionCall/thought part —
including non-signing models, where unsigned parts are the normal state.
session.metadata is persisted and pinned by the storage goldens, so the
per-turn counter drifted the google-wire state digest and red'd
tests/storage-engine-proofs.test.ts (google/keepFile: persisted state
drifts beyond the one-time ledger alignment vs committed goldens).

The counter only matters on a model that signs its calls (the only case
where unsigned parts are anomalous and the warn can fire), so gate the
increment on the modelSigns latch. Non-signing sessions no longer touch
the key; the storage goldens stay green. Verified: storage-engine-proofs
(google wire) + golden-storage-render + issue5253 audit suite all pass
locally on the fixed head.
Brings the branch from fc0c086 (2026-10-31) to the current ci-fixes
head so CI runs against the live master state instead of a 1416-commit-
stale base. Conflict in prepare-google.ts resolved by union of intent:
keep the upstream changes and the #5253 audit block (counter gated on
the modelSigns latch).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working dsh p1 scope/wiring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant