Skip to content

test: adversarial byte-fidelity corpus for tool-call arguments, all four wires - #2743

Merged
ranxianglei merged 1 commit into
masterfrom
2026-10-11_toolargs-corpus
Oct 11, 2026
Merged

ranxianglei merged 1 commit into
masterfrom
2026-10-11_toolargs-corpus

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Second deliverable of the consumer-path audit (#2738 follow-up; the remaining 🟡-grade gap from the audit table): point coverage for tool-argument byte fidelity becomes a systematic adversarial corpus.

What this pins

tests/toolargs-adversarial.test.ts — 674 cases crossing four directions × four wires × 13 hostile payloads × byte-level SSE chunkings:

Directions (every hop the bytes take):

Wires: chat, anthropic, responses (both real argument paths — ACP-tool delta accumulation AND client-tool done-carries-full passthrough), google (string args and object args).

Payloads (#1039 discipline — arguments are user intent): quote/escape soup, CRLF/tab escapes, CJK, astral 4-byte, literal \u sequences, tag-shaped bytes built with hex escapes, SSE-frame look-alikes, empty string, bare-empty args, whitespace-only, 2 KiB, unparseable-but-forwardable truncated JSON, duplicate keys + nesting.

Chunkings: whole / 1-byte / seeded RNG (17, 97) / every-byte-offset two-chunk splits for short payloads (splits inside escapes, inside multi-byte UTF-8, across frame terminators, mid-data:-prefix).

Assertion form: string wires (chat, responses) byte-exact; anthropic/google (parsed-object wires) canonical JSON equivalence, with google's documented {} degradation for unparseable args asserted as such.

Teeth (A/B)

A planted defect — "helpfully" normalizing arguments via JSON.stringify(JSON.parse(args)) at the chat passthrough emission site — turns 18 cases red (literal \u and duplicate keys payloads across all chunkings). Clean tree: 674/674.

Three harness bugs were found and fixed during development (worth recording): Buffer.toString() re-encoding destroying byte-split multibyte sequences (must enqueue raw bytes), template-literal \" silently corrupting corpus escapes (String.raw / JSON.stringify construction), and responses non-ACP tools relaying full arguments on output_item.done rather than deltas (both real paths now covered).

Config surface

None — test-only.

Kernel boundary

None — host tests only; kernel used read-only via its public wire exports.

…our wires (#1039, #1960)

Four directions (upstream parse / downstream emit / re-request rebuild /
client-echo round-trip) crossed with 13 hostile payloads and byte-level SSE
chunking (whole, 1-byte, seeded RNG, every-offset splits for short payloads).
String wires asserted byte-exact; anthropic/google object wires asserted
canonically. Teeth verified: a JSON-normalize defect planted on the exercised
chat passthrough emission site turns 18 cases red; clean tree 674/674.
@github-actions

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-11_toolargs-corpus (4e22b1a)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.2988-pr.2743.7409a7e

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr2743.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Review of PR #2743 (2026-10-11_toolargs-corpus @4e22b1a13, base master @e1cd9e11c) complete. Diff confirmed minimal: 1 commit, 1 added file tests/toolargs-adversarial.test.ts (+408/-0), no src/kernel/config/version touches. Both claims in the description were re-verified locally, not taken on faith.

Verification performed (targeted, local):

  • Clean tree: node --import tsx --test tests/toolargs-adversarial.test.ts → 674/674 pass, ~2.2 s (run twice, stable).
  • Teeth check: planted exactly the described defect — guarded JSON.stringify(JSON.parse(tc.arguments)) at the chat passthrough emission site (src/loop/adapter-openai.ts settle path, real-tool branch, line ~418) → exactly 18 failures: literal backslash-u sequences × 8 (frags 1/3 × whole/1-byte/rng-17/rng-97) + duplicate keys + nesting × 8 + 2 A4 every-byte-offset tests. Reverted; tree green again. Matches the PR's claim precisely.
  • Why only those 18: the other 11 payloads are canonical-form-invariant under re-stringification (escapes/CJK/astral/tag-shaped/SSE-shaped all round-trip byte-identical through parse→stringify) — the corpus is adversarial exactly where normalization defects bite. That is the design working, not a hole.
  • Harness bugs the PR says it fixed: confirmed present-and-correct in the final file — raw-Buffer chunking through ReadableStream (no string re-encoding of split multibyte UTF-8), String.raw/JSON.stringify payload construction, and responses covering BOTH real argument paths (ACP delta accumulation + client-tool done-carries-full).
  • Conventions: imports (../src/loop/index.ts, acp-kernel/wire), as never body casts, commit message format — all consistent with sibling suites. Deterministic (seeded LCG, no ports/network/env), auto-picked-up by the npm test glob.

Structured conclusion (八问):

  1. 是什么问题:tool-call 参数字节保真此前只有零散点覆盖,Consumer-path audit proofs: ledger drop-fraction + honest decompress failure for reaped blocks #2738 消费者路径审计把它列为剩余 🟡 缺口("tool-args adversarial corpus");本 PR 把它变成系统性对抗语料(4 方向 × 4 线 × 13 载荷 × 逐字节 SSE 切分)。
  2. 什么原因导致的:不是缺陷修复,是证明缺口——[bug] 插件透传会剥掉工具调用参数里的 render tag(arguments 流挂了 tag-echo 过滤器) #1039 纪律(arguments 即用户意图)+ [acp-loop] In-turn compress re-request edits the latest assistant message → Anthropic 400 "thinking blocks … cannot be modified" (session stays broken) #1960 危险类(re-request rebuild)要求四跳(parseStream / emitToolCall / buildRequest / kernel toCore)全部逐字节,而此前没有任何单一测试钉住这个矩阵。
  3. 影响范围与严重程度:纯测试,零运行时影响;合并后 CI 增加约 2 秒本地耗时。无用户面变化。
  4. 怎么修复的:语料矩阵化——敌意载荷(转义汤/CJK/星区码位/字面 \u/tag 形状/SSE 帧形似/空/不可解析截断/重复键)× 全字节偏移切分(转义内、多字节 UTF-8 内、帧终止符跨切、data: 前缀中间);断言形态按线语义校准(字符串线 byte-exact,对象线 canonical JSON,google 对不可解析参数的 {} 降级被明确断言为文档行为而非掩盖)。A/B 牙齿我实测成立(见上)。
  5. 是否回归问题:无。纯新增文件,既有行为清单为空(diff 未触碰任何 src/kernel/配置/持久化/线协议路径)。确定性已验证,无 flake 风险。
  6. 是否破坏缓存:不涉及(无运行时代码)。
  7. 是否改了设计方向:不改。kernel 只读使用公开 wire 导出。范围说明:原缺口措辞含 "retry × rollback",但重试与回滚都经由 buildRequest 从 core state 重建请求体(方向 C 已覆盖),这两条路径不存在独立的参数变更机制,故论证上覆盖完整。
  8. 是否引入新配置:无。

CI:guard 类 job(kernel-guard / version-guard / docs-coverage / lane-guard / docs-sync / pi-subagents-guard / lock-sync / unused-exports / config-surface-guard / build-artifact)全绿;4 个 test job 在我检查时仍在跑——本地定向验证已两次全绿,等 CI test job 转绿即可合。

判定:可合并(待 CI test job 转绿;合并按规矩由人来执行:#2743 )。review 中未发现需要顺手修的小问题。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Follow-up: PR #2743 head commit (4e22b1a) 全部检查已转绿——4 个 test job(ubuntu 22/24、windows 22/24)completed/success,加上此前已绿的 10 个 guard job,共 14/14。本地定向验证(674/674 + 植入缺陷 18 red)结论不变。可以合并:#2743

@ranxianglei
ranxianglei merged commit 43c1b57 into master Oct 11, 2026
14 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

📦 Merged Build Published

This merge to master was published as 0.0.267-master.43c1b57 (rolling master tag).

npm install -g billion-context@0.0.267-master.43c1b57  # this exact build
npm install -g billion-context@master  # always the newest merged build

npm view billion-context@master prints the current version at any time.

@ranxianglei
ranxianglei deleted the 2026-10-11_toolargs-corpus branch October 15, 2026 05:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant