Skip to content

[PR] feat: add I1/I2 outbound-invariant judge for cache-stability attribution - #3866

Closed
ranxianglei wants to merge 4 commits into
masterfrom
2026-10-20_invariant-judge
Closed

ranxianglei wants to merge 4 commits into
masterfrom
2026-10-20_invariant-judge

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Closes #3842

What this is

Implements the owner-approved route from #3842 (the corrected direction after the PBT/SMT layer was rejected): judge I1/I2 on real traffic + a general host-behavior generator, with the existing cache-proof scripts demoted to a regression archive.

Two machine-checkable invariants, reconciled on the live request path:

  • I1 (no fold) — every inbound message reappears at egress in order, core-identical, modulo tail growth plus a closed exemption list (chain carriers, acp_summary carriers, reasoning drops, crush envelopes). Any other mid-history difference = violation.
  • I2 (fold) — all divergence confined to [earliest fold coverage .. end); the frozen prefix is identical; the carrier position is pinned (drift = violation).

Why exit cores are re-parsed from the body

The outbound body is built from rebuiltMessages, not processedMessages (which is bookkeeping only). Re-parsing prepared.body through its own wire codec makes the comparison reflect exactly what crossed the wire, and unifies same-wire vs cross-wire through the kernel canonical core (#3658). Render-tag scaffolding (renderTags=text-only) is normalized away on both sides, so the exemption list closes without documenting a carve-out — no change to outbound bytes.

Enforcement split (deliberate)

  • CI enforcement: ACTIVE. tests/invariant-judge-e2e.test.ts arms the judge and drives the real proxy through a seeded host-behavior generator (append / edit-prior / retry / model-switch / restart). Legit stream → zero false positives; injected rewrite/reorder/drop/mid-insertion faults → all caught. These are the acceptance fixtures proving the generator + oracle find the shape family.
  • Production runtime: BUILT + WIRED + VERIFIED, but DISARMED by default (zero hot-path cost). Every request already calls maybeRunInvariantJudge; it is a no-op until armed. Arming on prod traffic is left as an owner decision because it trades CPU (a codec re-parse per request) against event volume, and it is the point at which a config knob would legitimately enter.

Config surface

No new config field, env var, or CLI flag is introduced. The judge is enabled only through the existing _set…ForTest idiom (tests). This keeps the change inside the owner-gated config-surface discipline — production enablement, if wanted, gets its own design pass first.

Behavior changes

None in production (advisory + disarmed). Only additions: one module, one guarded call site in src/server/handle.ts, two test files. The call site is guarded (prepared && protocol && !countTokens && !responsesCompact) and the judge never throws or mutates the body.

Regression review

Tests

  • tests/invariant-judge.test.ts — unit oracle: positive controls (identity, render-tag normalization, tail append, carriers, reasoning drop, crush envelope, tool-call id re-mint) + five-shape negatives + fold boundary. Green locally (targeted).
  • tests/invariant-judge-e2e.test.ts — real-proxy generator + defect fixtures. Green locally (targeted).
  • Full-suite verdict deferred to remote CI on this head (§7.2).

@ranxianglei ranxianglei changed the title feat: add I1/I2 outbound-invariant judge for cache-stability attribution [PR] feat: add I1/I2 outbound-invariant judge for cache-stability attribution Oct 19, 2026
@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-20_invariant-judge (f729104)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.4591-pr.3866.11b6357

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr3866.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

查重:不重复。 全仓 open+closed 按主题/失败签名检索(invariant judge / cache-stability / I1 I2 / outbound invariant):仅本 PR 命中;#3842 是其父设计 issue(owner 已拍板方向),不是并行实现;无其他在途 PR 做同一件事。

标签:pr + scope/wiring + p3 —— 功能新增非缺陷;判官挂出站 wire 边界(协议接线层);生产默认 disarmed、零行为变更 → p3。

本 PR 是 #3842 owner 批准路线(判官挂真实流量 + 通用宿主行为生成器,PBT/SMT 层已否决)的闭环交付,现在进入 review:对照批准方向核代码、逐项过既有行为回归清单、审测试有效性(真测试还是假测试)、查 pushed head 的 CI 状态,随后按八问体例给结论。

awork added 4 commits October 20, 2026 04:42
…und cores (#3842)

Reconciles inbound vs outbound message cores on the live request path to enforce two cache-stability invariants machine-checkably: I1 (no fold) requires every inbound message to reappear at egress in order, core-identical, modulo tail growth plus a closed exemption list (chain carriers, acp_summary carriers, reasoning drops, crush envelopes); I2 (fold) confines all divergence to [earliest fold coverage .. end) with the frozen prefix identical and the carrier position pinned.

Exit cores are re-parsed from the exact outbound body through its own wire codec, so the comparison reflects what crossed the wire and is wire-neutral (same-wire and cross-wire unified via the kernel canonical core). Render-tag scaffolding (renderTags=text-only) is normalized away on both sides, so the exemption list closes without documenting a carve-out.

The judge is advisory and disarmed by default (zero hot-path cost); it is armed in tests today, and production arming on real traffic is deliberately left as an owner decision (perf/event-noise tradeoff and the point at which a config knob would enter). CI enforcement is active: a seeded host-behavior generator (append/edit/retry/model-switch/restart) drives the real proxy through the judge with zero false positives, and injected rewrite/reorder/drop/mid-insertion faults are all caught.

Tests: unit oracle (positive controls + five-shape negatives + fold boundary) and a real-proxy e2e generator + defect fixtures. Fixes nothing upstream; the referenced defects (#3805/#2483/#3691/#3769/#3676) stay scheduled separately per the issue.
…entity (#3842)

Review findings on the I1/I2 judge (PR #3866):

- The call site sat right after runPrepare(), BEFORE preflight compression.
  A fold-CREATING request was judged on its pre-fold bytes, so I2 never
  reconciled the one traffic class it exists for. Move the call to the
  resolved prepared value (after preflight / reforge), where prepared.body
  is the exact byte stream relayed upstream.

- The scan skipped every exit carrier while the entry side only skipped
  reasoning: a stateful host that re-sends previously-injected acp_summary /
  chain-checkpoint carriers false-positived on every steady-state post-fold
  turn (even entry==exit). Carrier entries now match their own twin by
  identity, and the gap budget counts non-carrier elements only.

- The sink delivered violations only, so folded-but-clean verdicts were
  invisible and the live I2 path could not be asserted. Every judged
  request now reports its verdict (test-only surface; prod stays disarmed).

Tests: unit fixtures for carrier echo (summary + chain), five-consecutive
carrier gap, and the carrier-text-drift design boundary; e2e gains a real
preflight fold through the live path (tiny window) - zero false positives
across the folded stream, and post-fold prefix corruption fires.
Pre-existing on master (introduced by #3811): the function has no importers
outside its own file, so the CI unused-exports gate fails on every head that
reaches it - which skips all downstream test lanes. In-scope fix for this PR's
merge commit to go green.
Pre-existing on master: the generated table still carried a superseded
storage.unified row (pre-#3818 'unset (shadow)' default) plus an older
sqliteAutoDownload row missing the #3788 cooldown text. The generator now
emits only the current rows; docs-sync gate passes again. No config surface
change - regenerated output only.
@ranxianglei
ranxianglei force-pushed the 2026-10-20_invariant-judge branch from 7a08a0a to f729104 Compare October 19, 2026 21:30
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

中间状态(非最终结论):复审发现 2 个真实缺陷 + 2 个阻塞 CI 的既有问题,已在本地修复并验证;push 排队等 GitHub egress 恢复。

  1. judge 放点过早(真实缺陷):原调用点在 runPrepare() 之后、preflight 压缩之前——折叠生成请求比的是折叠前的旧字节,I2 在最关键的路径上没真正生效。移到 prepared = outcome; 之后,比对实际出网的最终字节。
  2. carrier echo 误报类(真实缺陷):扫描对出站侧跳过所有 carrier、入站侧只跳 reasoning——有状态 host 逐字回显之前注入的 acp_summary/chain checkpoint(稳态下 inbound==outbound)时每轮都误报。改为按身份匹配(carrier 条目找字节一致的孪生),gap 窗口只数非 carrier 元素。单测补 4 个 fixture(含 1 负向)。
  3. CI 红 = master 既有的两个 gate 失败(纯 master worktree 验证过):unused-exports(本 PR 4 个内部类型→去 export;master 自身还有一处 maskUrlForPublish)与 docs-sync(CONFIGURATION 陈旧 storage.unified 行)。在本分支以独立标注 commit 修掉——否则 merge commit 同样红、test lane 持续被跳过。
  4. 补上 scope gap:[design] Pin cache stability with two outbound invariants: no-fold ⇒ exit only appends a tail; fold ⇒ bytes before the f | [design] 用两条出站不变量定死缓存稳定:无折叠=出口只多尾部,有折叠=折叠点前字节不动(证明式断言 + 属性/SMT 搜索分层) #3842 批准方案里生成器动作集含「触发折叠」,原 e2e 从未触发真实折叠(I2 没走过 live path)。新增小窗口场景:真 proxy 真 preflight 折叠,断言整条流(含折叠生成轮与后续稳态)零误报,并注入故障证明检出力。sink 契约改为投递全部 verdict(原仅 violations),让 folded-but-clean 流量可观测(仅测试面)。

本地验证:定向测试 20/20 绿、tsc 干净、两个 gate 干净。全量套件按 §7.2 交给 CI。生产行为零变化:judge 默认仍 disarmed,无新配置面。

当前阻塞:GitHub git egress 暂时不可达(API 侧正常),后台重试 push 进行中;push 成功后补八问结论 + CI 结果。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

(ai: vllm-qwen/qwen3.8-27b)

复审完成:原实现有 2 处「接线了但没生效」的真实缺陷 + 1 处范围缺口,已全部修复并推送(f729104e3,rebase 到当前 master;整体 CI 绿)。八问结论:

  1. 是什么问题:[design] Pin cache stability with two outbound invariants: no-fold ⇒ exit only appends a tail; fold ⇒ bytes before the f | [design] 用两条出站不变量定死缓存稳定:无折叠=出口只多尾部,有折叠=折叠点前字节不动(证明式断言 + 属性/SMT 搜索分层) #3842 要的是一台机器可校验的出站不变量裁判(I1 无折叠 / I2 有折叠),用于把缓存断裂归因到宿主行为还是 bili 自身。原 PR 有三处实质缺陷:①judge 调用点在 preflight 压缩之前,折叠生成请求比的是折叠前旧字节,I2 在最关键路径上从未真正生效;②扫描对出口侧跳过所有 carrier、入口侧只跳 reasoning——状态保持型宿主把注入过的 acp_summary/chain checkpoint 逐字回显(稳态下入口==出口)时每轮都误报;③e2e 生成器从未触发过真实折叠(I2 没走过 live path),与 [design] Pin cache stability with two outbound invariants: no-fold ⇒ exit only appends a tail; fold ⇒ bytes before the f | [design] 用两条出站不变量定死缓存稳定:无折叠=出口只多尾部,有折叠=折叠点前字节不动(证明式断言 + 属性/SMT 搜索分层) #3842 批准方案的动作集(追加/编辑中历史/触发折叠/换 id 重启/切模型/重试)不符。
  2. 什么原因导致的:①调用点落在「prepare 完成」而非「最终出站字节」(preflight 可能返回新的出站对象);②豁免清单按「种类跳过」设计而不是按「身份匹配」——carrier 本来就是代理注入进历史的消息,宿主有权逐字回显,按种类跳过把「回显」当成「篡改」;③生成器动作集实现不完整。
  3. 影响范围与严重程度:生产行为零影响——judge 默认仍 disarmed(热路径零成本)、无新增配置。缺陷严重度属「特定条件下能力不工作」:若照原样在生产启用,①会漏掉最该归因的折叠类缓存断裂,②会在状态保持型宿主流水线上刷出假阳性事件风暴。非用户可见故障。
  4. 怎么修复的:①调用点移到 prepared = outcome; 之后(最终将被 relay 的字节),guard 条件不变;②carrier 改为身份匹配——carrier 条目在出口侧找逐字相同的孪生(stableEqual),gap 窗口只计非 carrier 元素(原先连续 carrier 超窗也会误报);并把设计边界写进单测夹具:carrier 原位文本漂移 = 绿(折叠点之后的内容在 bili 自有渲染区,宿主对其编辑会被 canonical 重渲染归一化,不归因于宿主);③e2e 新增小窗口场景:真实代理真实触发 preflight 压缩,断言含折叠生成轮与后续稳态轮全流零误报,再向冻结前缀注入损坏证明检测力;sink 契约从「仅违规」改为投递全部 verdict(仅测试面,生产 sink 为 null),folded-but-clean 流量从此可观测。
  5. 是否有回归问题:逐项审过既有行为——handle.ts 调用点带 guard(prepared && protocol && !countTokens && !responsesCompact),disarmed 时一个布尔检查即返回,生产路径/时序/输出无变化;judge 只读 core、从不写回 body/processedMessages,wire fidelity 未动;两种压缩模式都推演过(plugin 模式 preflight block 保留 acp_summary 锚点,I2 可识别;agent 发起的 plugin 模式折叠走 compress tool-call,judge 不识别其为 summary carrier——已记录为边界,本轮不扩大)。全量套件裁决:CI 6/6 test shard 绿、全部 static gate 绿、全部 e2e lane 绿(见下)。
  6. 是否破坏缓存:否。judge 纯读,不改出站任何字节;render-tag 归一化只在比较内部进行,不在 wire 上。缓存账本/去重键/增量游标均未触碰。
  7. 是否改了设计方向:否。严格按 [design] Pin cache stability with two outbound invariants: no-fold ⇒ exit only appends a tail; fold ⇒ bytes before the f | [design] 用两条出站不变量定死缓存稳定:无折叠=出口只多尾部,有折叠=折叠点前字节不动(证明式断言 + 属性/SMT 搜索分层) #3842 owner 批准的路线执行(真实流量 + 通用宿主行为生成器,旧 cache-proof 脚本降级为回归档案),未引入被否决的 PBT/SMT 层。
  8. 是否引入了新的配置:无新字段/env/CLI(守 owner-gated config-surface 纪律)。judge 目前只经 _set…ForTest 测试缝启用。是否在生产启用(CPU 换归因事件量、是否因此需要配置旋钮)留作 owner 决策——这是唯一未决项。

CI 与附带处置:

判定:可合并。代码、测试、CI 全部闭环;唯一遗留决策项是生产是否启用 judge(建议暂缓,先小范围武装观察事件量与 CPU 开销再定)。

Zhishulo pushed a commit to Zhishulo/billion-context that referenced this pull request Oct 20, 2026
…entity (ranxianglei#3842)

Review findings on the I1/I2 judge (PR ranxianglei#3866):

- The call site sat right after runPrepare(), BEFORE preflight compression.
  A fold-CREATING request was judged on its pre-fold bytes, so I2 never
  reconciled the one traffic class it exists for. Move the call to the
  resolved prepared value (after preflight / reforge), where prepared.body
  is the exact byte stream relayed upstream.

- The scan skipped every exit carrier while the entry side only skipped
  reasoning: a stateful host that re-sends previously-injected acp_summary /
  chain-checkpoint carriers false-positived on every steady-state post-fold
  turn (even entry==exit). Carrier entries now match their own twin by
  identity, and the gap budget counts non-carrier elements only.

- The sink delivered violations only, so folded-but-clean verdicts were
  invisible and the live I2 path could not be asserted. Every judged
  request now reports its verdict (test-only surface; prod stays disarmed).

Tests: unit fixtures for carrier echo (summary + chain), five-consecutive
carrier gap, and the carrier-text-drift design boundary; e2e gains a real
preflight fold through the live path (tiny window) - zero false positives
across the folded stream, and post-fold prefix corruption fires.
Zhishulo pushed a commit to Zhishulo/billion-context that referenced this pull request Oct 20, 2026
…ontinuity; bound the exemption list (ranxianglei#3938 PR-A)

- Arm by default (ranxianglei#3938 decision 2): violations log at warn, clean verdicts at
  debug, never blocks. Knob diagnostics.invariantJudge / BILI_INVARIANT_JUDGE=0
  to disarm (CONFIGURATION en+zh).
- Move the main-lane hook from the pre-send site (handle.ts) to the universal
  egress chokepoint (relay.ts noteForwardedBody block): judges the EXACT bytes
  after compat-role / steering / chain-stamp / shed rewrites.
- New loop-lane continuity check (loop/core.ts fetchUpstream): rounds 2+ have
  no client inbound, so I1's entry ground truth is the previous outbound;
  clipped/oversized forensics silently skip.
- ranxianglei#3842 count-bounded exemption list: carrier/reasoning/crush volume rides
  every verdict (exempt ledger) and beyond the cap (128 default) is itself an
  exemption-overflow violation.
- e2e now proves all three wires live (openai + anthropic + google): zero
  false positives on legitimate growth, teeth on injected mid-history defects.

Rebased from ranxianglei#3866 (its 2 content commits ride as the first two commits of
this branch).
@ranxianglei

Copy link
Copy Markdown
Owner Author

(ai: zhipuai-lb/glm-5.3)

#3866 判官已以更强形态落地:rebase 后作为 PR #3944 转正(L1 advisory 默认全量开 + 主/loop 双车道挂点 + 豁免计数封顶 + 三线 e2e),豁免闭合续作 #3985。原分支停止维护,作为被取代方关闭。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

1 participant