Skip to content

test: wire-contract anti-regression system — parity fakes, goldens, unified ledger - #1305

Merged
ranxianglei merged 6 commits into
masterfrom
2026-09-25_wire-contract-gates
Sep 25, 2026
Merged

ranxianglei merged 6 commits into
masterfrom
2026-09-25_wire-contract-gates

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

What

Implements owner-decoped items 1+2+3 of #1304 plus the phase-(a) scripted-client matrix (4 protocols × proxy mode) — pure test/CI infrastructure, zero runtime code changes, no release-workflow changes.

Item 1 — validation-parity fake upstreams (tests/wire-contract-fakes.ts)

Item 2 — golden schema snapshots (tests/golden/wire-contract/*.json)

Byte-equal snapshots of every advertised tool schema at both layers: kernel exports (all arrays + individual constants incl. optional absorb/image_full/retrieve variants), bili-built surfaces (BILI_ACP_TOOLS_*, search_context/decompress extensions, RULE_TOOL*, retrieveToolsFor, IMAGE_FULL_TOOL_GOOGLE), and the plugin manifest (default + opt-in). Manifest version is normalized to <VERSION> so release commits don't churn it. Drift fails with instructions; regeneration is explicit: node --import tsx scripts/update-wire-contract-goldens.ts, with justification required in the PR that changes a snapshot.

Item 3 — unified gate + institutional rule (tests/wire-contract.test.ts, AGENTS.md §7.3)

Consolidates the #1302 kernel-export gate (tests/kernel-pin-wire-schema.test.ts) and the #1301 served-surface gate (tests/issue1299-anthropic-top-level.test.ts) into one module — both files are deleted here and every assertion they made is preserved (layer A = kernel exports incl. the 'Do NOT pin this kernel version' failure text; layer B = bili surfaces incl. the compress call-form property check; layer C = manifest; plus a ledger self-check asserting every rule id has a live enforcement clause). AGENTS.md §7.3 gains the permanent rule: every upstream rejection/validation constraint discovered in production or docs becomes a wire-contract rule inside the fixing PR; the ledger only grows.

Behavior changes

None. No runtime code touched; no config/persistence/wire-format changes. The only observable effect is new CI signal: future pin bumps or tool-surface changes that emit a wire-illegal schema now fail npm test instead of reaching production.

Pre-flight (on current master, dc8bdea)

  • npm run typecheck ✅
  • tsc --noEmit over the new test files: clean (repo-wide root-tsconfig noise in pre-existing test files is untouched)
  • npm test: 2569 tests, 0 failures ✅
  • npm run build ✅
  • Gates run in the normal node --import tsx --test tests/*.test.ts glob, so every PR (incl. pin-bump PRs) inherits them automatically.

…nified ledger (#1304)

Consolidates the #1301 served-surface gate and the #1302 kernel-export gate
into one wire-contract suite and extends it per owner decision on #1304
(items 1+2+3, plus the 4-protocol scripted-client matrix phase):

- tests/wire-contract-fakes.ts: WIRE_RULES ledger (WC-001..WC-006, each with
  production/doc provenance) + strictest-known validation for all four wires
  (anthropic /v1/messages, openai chat, responses, google v1beta) + four
  validation-parity fake upstreams on listen(0) with canned per-wire replies.
- tests/wire-contract.test.ts: four layers — ledger self-check (every rule id
  has a live enforcement clause), A kernel exports, B bili-built surfaces,
  C plugin manifest (default + opt-in), D live proxy-mode forward of one
  client-shaped turn per protocol through the matching fake (asserts the
  forwarded tool set is exactly bili's intended surface and the fake's
  strictest validation accepts it). The anthropic lane reproduces #1299
  end-to-end. Supersedes tests/kernel-pin-wire-schema.test.ts and
  tests/issue1299-anthropic-top-level.test.ts (all their assertions kept).
- tests/wire-contract-golden.test.ts + tests/golden/wire-contract/*.json:
  byte-equal golden snapshots of every advertised tool schema at both layers
  (kernel exports, bili surfaces, plugin manifest default+opt-in); manifest
  version normalized to <VERSION> so release commits don't churn it.
- scripts/update-wire-contract-goldens.ts: explicit regeneration path.
- AGENTS.md §7.3: institutional rule — every upstream rejection/validation
  constraint becomes a permanent wire-contract rule inside the fixing PR;
  the ledger only grows.
@github-actions

github-actions Bot commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-09-25_wire-contract-gates (d75af8a)

Option A — Install from npm PR tag (recommended)

npm install -g billion-context@pr-1305

Each push to this PR publishes a new version under the pr-1305 npm tag.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr1305.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Review started. Pulling branch 2026-09-25_wire-contract-gates (81befc6, single commit on current master dc8bdea). Plan per review spec: ① direction check against #1304 → ② regression audit incl. verifying every assertion from the two deleted gate files survives in the unified module → ③ code review of fakes/inventory/gates → ④ test-tooth check → ⑤ diff cleanliness + CI status. Full conclusion to follow shortly.

ework-agent added 3 commits September 25, 2026 14:28
…autocrlf)

CI run 36102348905: all 4 golden tests failed on windows-latest because GitHub Actions checkouts materialize committed-LF files as CRLF in the worktree; runtime-generated canonical text uses LF. Normalize disk content to LF so the gate is platform-deterministic while still catching any real content drift.
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 ## Review verdict: mergeable — CI fully green on 64952f9d (10/10 checks)

Reviewed against current master (dc8bdea5, zero lag). Per the four gates in §7.2:

① Direction (vs #1304) — PASS

Scope matches the owner-decoped items exactly: 1+2+3 + phase-(a) matrix, pure test/CI infra, zero runtime changes, deterministic listen(0) fakes, gates inside the normal npm test glob so every future pin-bump PR inherits them. No release-workflow or runtime surface touched.

② Regression audit — both deleted gates preserved assertion-by-assertion

I recovered both superseded files from dc8bdea5 and walked every assertion into the new module:

  • tests/kernel-pin-wire-schema.test.ts (fix: pin acp-kernel 0.0.89 — Claude-lane hotfix (top-level anyOf removed from compress schema) #1302): all 3 tests covered — Layer A keeps the same 5 kernel arrays, the combinator scan (plus not), the 'Do NOT pin this kernel version' failure text, and the 9 individual/optional constants (now asserted present instead of skip-on-undefined — strictly stronger). Its conditional pluginManifest/buildPluginManifest export validation was dropped because the pinned kernel exports neither symbol and src/ never references them — it was already a no-op; Layer C now validates bili's actual served surface (handlePluginManifest) instead. No loss.
  • tests/issue1299-anthropic-top-level.test.ts (fix: pin acp-kernel 0.0.89 — plain-top compress input_schema restores Claude requests (#1299) #1301): served-surface walk → Layer B (incl. the compress call-form property check), manifest fetch → Layer C (direct capture, default + opt-in, base-presence lists), live proxy forward → Layer D (200 round-trip, exactly one upstream POST, zero validator violations, forwarded tool set exactly base+client tools — stronger than the old contains-checks). The old test's ABSORB_TOOL/IMAGE_FULL_TOOL were kernel re-exports (src/compress-tool.ts:219), covered by Layer A + kernel.json.
  • One nuance, noted not blocking: explicit top-level-combinator clauses exist for anthropic (WC-001) and google (WC-006) only; openai-chat/responses validators check name pattern + parameters.type=="object". The byte-equal goldens backstop any schema change regardless, and the issue itself says "start documented, grow via ledger" — add WC rules there when/if such a constraint is ever observed.
  • Existing behavior inventory conclusion: no regressions. Only observable delta is new CI signal, as stated.

③ Code review — clean

Fakes bind loopback only on listen(0); requests[] counts POSTs only (GET /v1/models handled first), so "exactly one upstream request" has teeth; every Layer D lane closes proxy+fake in try/finally; goldens contain no timestamps/random ids and normalize manifest version to <VERSION>; orphan-file guard pins the directory to the tracked set. Local runs: 16/16 gate tests pass, stable on rerun.

④ Test-tooth check — verified, not assumed

Simulated the #1299 payload shape (input_schema with top-level anyOf) through validateAnthropicBody → returns WC-001 tools[0] (compress): top-level "anyOf" rejected by Anthropic (#1299); current BILI_ACP_TOOLS_ANTHROPIC surface → zero violations. The ledger self-check probes every rule id through its live enforcement clause. These are real tests: the bug this suite exists for would turn them red.

⑤ Diff cleanliness — two issues found, both fixed directly on the branch

  1. AGENTS.md indentation drift (be943ca0): the §7.3 edit re-indented two pre-existing bullets ("Symptom ≠ mechanism", "Honest output") from column 0 to 1-space along with their continuation lines. Renders identically in CommonMark, but pre-existing lines shouldn't churn — restored them to their exact original indentation; the new ledger bullet now matches the standard list style.
  2. Windows CI failure — real bug, caught by CI: on head 81befc6d, both test (windows-latest, *) jobs failed with exactly the 4 golden byte-equal tests (master's Windows jobs were green; these files don't exist there). Root cause, proven both ways: the repo has no .gitattributes, committed blobs are pure LF, and Git-for-Windows' default autocrlf=true materializes them as CRLF in the worktree → actual (in-memory, LF) ≠ expected (disk, CRLF) — visible verbatim in the win24 log. A control clone of the pre-fix commit with autocrlf=true reproduces CRLF for all 4 files; with the fix, LF. You'd already pushed 4df94de6 (normalize disk content before compare — good, keeps the gate platform-deterministic); I stacked the root-cause fix on top (64952f9d): .gitattributes with tests/golden/wire-contract/*.json text eol=lf, scoped to exactly those paths, no other file affected (git check-attr verified). The two mechanisms are complementary — attributes keep working copies in canonical form, your normalization keeps the gate green even if the attribute is ever removed. If you prefer minimalism, dropping either one still leaves a passing, drift-catching gate; my recommendation is to keep both.

Post-fix CI on 64952f9d: 10/10 success, including test (windows-latest, 22) and (24).

Behavior changes: none (rule 4 checked — no threshold/default/timing/format drift; test/CI signal only). Not merged — that part is yours: #1305

中文摘要:审查了 wire-contract 测试体系 PR——方向符合 #1304 决议、两个被删旧门禁的断言逐条保留且更强、测试有牙(模拟 #1299 负载确实触发 WC-001);发现并直接在分支上修了两个问题:AGENTS.md 既有列表项的缩进漂移,以及 Windows CI 上 golden 字节比对因 autocrlf 转 CRLF 必挂的确定性 bug(补了 .gitattributes 强制 LF,与你已推的归一化修复互补),修复后 10/10 CI 全绿,可以合并。

…enai SSE lane

Review of #1305 (mutation-verified) found the consolidation dropped the
old #1302 assertion for the OpenAI chat and Responses tool shapes:
top-level combinators were only checked on anthropic (WC-001) and google
(WC-006), so an anyOf on COMPRESS_TOOL_OPENAI.parameters passed every
rule test and only the golden byte-diffs caught it. Restores the ban as
part of WC-004/WC-005 with provenance; re-running the mutation now fails
6 rule tests (was 0).

Also: stream:true on the openai-chat smoke lane (activates the fake's
SSE branch and exercises the real src/stream-openai.ts parse path —
previously dead code), sse() no longer emits an empty event: line for
OpenAI chunks, and AGENTS.md now points WIRE_RULES at
tests/wire-contract-fakes.ts (where it is defined).
@ranxianglei

Copy link
Copy Markdown
Owner Author

Independent deep review + fixes (head dc0f59d)

Review method: throwaway worktree, mutation testing against the historical bug shapes, golden idempotence checks, deleted-file assertion diffing.

✅ Headline — the system bites

🔴 Found & fixed: one dropped assertion (mutation-verified)

The consolidation lost the old #1302 combinator ban for the OpenAI chat / Responses shapes — combinators were only checked on anthropic/google. Mutation C (anyOf on COMPRESS_TOOL_OPENAI.parameters) passed all 11 rule tests; only golden byte-diffs caught it, and a dev who "fixed" CI by regenerating goldens would lose the guard silently.

Fix in dc0f59d: ban restored as part of WC-004/WC-005 with provenance. Mutation C now fails 6 rule tests (re-verified, then reverted — 11/11 green).

Minor fixes in the same commit

  • openai-chat lane now sends stream: true — activates the fake's dead SSE branch and exercises the real src/stream-openai.ts parse path end-to-end
  • sse() no longer emits an empty event: line for OpenAI chunks (real OpenAI SSE has none)
  • AGENTS.md now points WIRE_RULES at tests/wire-contract-fakes.ts (where it's actually defined)

Noted, not blocking (follow-ups)

  • Phase (a) only: no plugin-mode (x-bili-plugin) lane, no CI workflow changes — honest in the commit message; phase (b) can layer on top
  • Manifest gate calls handlePluginManifest directly (any future HTTP-route filtering is invisible to it)
  • Golden regen is green-by-construction; cheap hardening = CI step update-wire-contract-goldens.ts && git diff --exit-code

Gates on dc0f59d: typecheck ✅ · 2559 pass / 0 fail (2569 tests) · build ✅. Ready for merge.

Symmetry with the WC-004/WC-006 probes: without a combinator-shaped
responses body, deleting the restored WC-005 top-level-combinator loop
would go undetected by the suite (the other two responses probes hit
the name/type clauses only).
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Round-2 review of dc0f59d — all claims independently verified; one one-line gap fixed directly on the branch (d75af8a1). Verdict: mergeable.

✅ Your finding is correct — and my round-1 call was too lenient

I re-derived the old #1302 gate's exact semantics from the deleted file (schemaOf: anthropic→input_schema, google→parameters, else tool.parameters ?? tool.function?.parameters) and confirmed the restored clauses check the same locations per shape (chat → function.parameters, responses → flat parameters). I then reproduced Mutation C myself against the new validators:

  • anyOf injected into K.COMPRESS_TOOL_OPENAI.function.parameters → WC-004 tools[0] (compress): top-level "anyOf" not portable…; oneOf/allOf/not each CAUGHT
  • same mutation on COMPRESS_TOOL_RESPONSES.parameters → caught by WC-005
  • unmutated kernel exports → zero violations (no false positives)

And because Layers A/B/C/D all route through the same VALIDATORS map, the mutation now fails at the rule layer (kernel exports → bili surfaces → both manifests → live openai-chat lane) with goldens uninvolved — exactly the silent-loss scenario your "dev regenerates goldens" argument describes. One honest correction to my own record: in round 1 I classified this gap as "nuance, noted not blocking" because goldens backstop it. Your mutation proof shows why that was wrong — under the item-3 every assertion preserved mandate, a ban the old gate had on all four shapes must be enforced on all four shapes by the rule layer itself. It isn't just defensible that you fixed it; it was required.

✅ Minor fixes audited, all sound

  • stream: true on the openai-chat lane: activates the previously dead SSE branch in openAiChatReply, so the lane now exercises the real src/stream-openai.ts parse path end-to-end. Good catch that it was dead weight before.
  • sse() empty-event skip: scope-verified. Anthropic (named events) and Responses (named response.* events) pass non-empty names → byte-identical output; Google never used sse() at all (googleReply writes raw data-only frames, which matches real Gemini v1beta); only OpenAI chunks change — and dropping the spurious event: line makes the fake more faithful to real OpenAI SSE. No lane regressed.
  • AGENTS.md pointer: genuinely wrong in the original text — WIRE_RULES is defined in tests/wire-contract-fakes.ts, not wire-contract.test.ts. Fixed correctly.

🔧 One small fix I pushed directly (d75af8a1)

The ledger self-check probes had a combinator-shaped body for openai-chat and google but not for responses — so deleting the freshly restored WC-005 combinator loop would go undetected by the suite (the two existing responses probes hit only the name/type clauses; the self-check keys off rule-id, not clause). Added the symmetric one-line probe: { tools: [{ type: "function", name: "ok", parameters: { type: "object", properties: {}, anyOf: [] } }] }. Suite green after.

Gates

  • Local: wire-contract suites 16/16 on dc0f59d; 11/11 after the probe commit
  • CI on d75af8a1: 10/10 success (incl. both windows-latest test jobs)

Your three "noted, not blocking" follow-ups (phase-(b) plugin-mode lanes, manifest-gate bypassing future HTTP-route filtering, golden-regen green-by-construction + suggested CI git diff --exit-code step) are all reasonable to defer — the last one especially is cheap insurance worth picking up whenever phase (b) lands.

No runtime code touched, diff stays test/docs-only, both deleted gates' assertions now provably preserved clause-for-clause. Ready for you to merge — I can't do that part.

中文摘要:独立复现了变异 C(OpenAI/Responses 形状的顶层组合子禁令确实曾在合并中丢失,已按 #1302 原语义恢复并经四层规则验证),另直接补了一行 ledger 自检探针堵住 WC-005 子句可被静默删除的对称缺口,CI 10/10 全绿,可以合并。

@ranxianglei
ranxianglei merged commit a713738 into master Sep 25, 2026
10 checks passed
ranxianglei pushed a commit that referenced this pull request Oct 10, 2026
…ate on Windows CI

The two Windows test failures are pre-existing on the base 98fb91c (run
38034641775 fails the identical assertion): tests/issue2605-dsh-retry-patch.test.ts
byte-pins '- id: compaction-basic\n  config:\n    auto: false\n' against the
repo-checked-out dsh.bundle.patch.yml, and Git-for-Windows' default
autocrlf=true converts it to CRLF at checkout, so includes() fails. The
runtime-written overlay lane (test 2) is LF by construction and passes.

Exactly the #1305 class: goldens got 'text eol=lf' for this; the yml was
never covered. One attribute line, no behavior change on Linux.
ranxianglei pushed a commit that referenced this pull request Oct 10, 2026
…2605)

tests/issue2605-dsh-retry-patch.test.ts reads the patch file raw and
byte-matches multi-line rows; Git-for-Windows autocrlf=true converts it to
CRLF at checkout, failing both windows CI shards on every run since #2610
landed (master red at 98fb91c, run 38034641775). Same mechanism as the
#1305 golden pin above. In-passing fix carried on #2625.
YunzhouLi-hub pushed a commit to YunzhouLi-hub/billion-context that referenced this pull request Oct 10, 2026
…t includes (ranxianglei#2631)

ranxianglei#2605's bundle test asserts byte-exact multi-line literals against the
shipped dsh.bundle.patch.yml; Git-for-Windows checks it out CRLF
(autocrlf) and the includes fail on every Windows leg since ranxianglei#2610
merged — master's test (windows-latest, 22|24) legs have been red
since 07:30Z, blocking every PR's required checks (ranxianglei#2631).

Same failure class as ranxianglei#1305 (golden JSONs): pin eol=lf in
.gitattributes, and belt-and-suspenders normalize CRLF after read in
the test so checkouts that bypass gitattributes still pass.
@ranxianglei ranxianglei mentioned this pull request Oct 10, 2026
@ranxianglei
ranxianglei deleted the 2026-09-25_wire-contract-gates branch October 15, 2026 05:32
aolshevskiy added a commit to aolshevskiy/billion-context that referenced this pull request Oct 17, 2026
@github-actions github-actions Bot mentioned this pull request Oct 17, 2026
ranxianglei pushed a commit that referenced this pull request Oct 17, 2026
…3484)

Same failure class as #1305/#2631/#2489 already pinned in this file:
Git-for-Windows' default autocrlf=true checked tests/golden/web-hitrate/
small-full.svg out CRLF, so the byte-equal golden gate failed every
windows-latest run (job 117352125307, test shard 22-1) while linux passed.
ranxianglei pushed a commit that referenced this pull request Oct 17, 2026
…3484)

Same failure class as #1305/#2631/#2489 already pinned in this file:
Git-for-Windows' default autocrlf=true checked tests/golden/web-hitrate/
small-full.svg out CRLF, so the byte-equal golden gate failed every
windows-latest run (job 117352125307, test shard 22-1) while linux passed.
suyeku1 added a commit to suyeku1/billion-context that referenced this pull request Nov 11, 2026
…anxianglei#1295)

windows-latest shard 4 (run 43029697326) failed one sub-test:

  not ok 2458 - trae plugin pipe: a clean captured turn is relayed
                byte-verbatim (no synthesized error)

The assertion is `assert.equal(out, REAL_TURN)` where REAL_TURN is
tests/fixtures/trae-2026-11-04-two-tool-calls.sse.txt read straight off disk.
The committed blob is LF, but .gitattributes had no rule for this path, so
Git-for-Windows' autocrlf=true checked it out CRLF. The pipe normalizes SSE
terminators to LF (src/sse-util.ts:normalizeSseLineEndings), so on Windows the
comparison was LF-output vs CRLF-fixture: every single line differed, which is
why the assert dumped the whole 9.6 KB turn as a 90-for-91 line replacement.

Pinned the fixture to eol=lf, the same remedy the repo already applies to
tests/golden/wire-contract/*.json (ranxianglei#1305), dsh.bundle.patch.yml (ranxianglei#2631),
tests/golden/web-trajectory/*.svg (ranxianglei#2489), tests/golden/web-hitrate/*.svg
(ranxianglei#3484) and tests/fixtures/vertex-anthropic/* (run 38982246862).

No content change: the blob was already LF, so this only fixes the checkout.
Linux and macOS legs were green throughout; nothing else moved.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant