Skip to content

kernel: teach the strict-JSON serialization contract in the canonical compress copy (#2587) - #2622

Closed
ranxianglei wants to merge 2 commits into
masterfrom
2026-10-10_kernel-strict-json-compress
Closed

ranxianglei wants to merge 2 commits into
masterfrom
2026-10-10_kernel-strict-json-compress

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

kernel-change: this PR modifies kernel/** (canonical compress copy). Kernel version bumped 0.0.112 → 0.0.113; wire-contract goldens regenerated via scripts/update-wire-contract-goldens.ts; both test layers green (npm --prefix kernel test 1284/1284, affected root suites + tsc clean). Implements #2587 (owner-directed: closes the interim-host-override path in favor of the canonical fix — #2588 was closed unmerged by owner decision). Root cause evidence: #2579, #2580.

What changes

The kernel-default compress surface framed the string form as "plain text … no JSON structure, no escaping" with unquoted ref-header examples ('m00150–m00220 optional topic'). Models then streamed content values that are not valid JSON — ~4% of compress calls on strict-JSON hosts: bare unquoted values with real newlines (A-class), or unescaped quotes inside strings (B-class) — and the host rejects the tool input at message_stop, failing the whole turn before any lenient parsing runs. On the Anthropic lane the tool channel is the only channel (text-marker parsing is responses-only), so the tool description's advice was actively harmful there.

Seven canonical spots now state the serialization contract explicitly:

Spot Change
kernel/src/compress-tools.ts COMPRESS_PARAMETERS.properties.content.description strict JSON rule up front: every string quoted; internal " written \", newline written \n; array (PREFERRED) vs ONE-string both described as JSON values
kernel/src/compress-tools.ts line-form item "verbatim (no JSON escaping)" → "still a quoted JSON string value: internal double quotes written \", newlines written \n"
kernel/src/compress-tools.ts COMPRESS_TOOL.description strict-JSON contract first line; PREFERRED flips to object-array with pasteable skeleton {"content":[{"startId":"m00122","endId":"m00127","summary":"...","topic":"..."}]}; string form kept but described as a quoted JSON string; flat form unchanged
kernel/src/compress-tools.ts system-prompt TOOLS section example flips to the array skeleton; string form kept with explicit in-quotes framing
kernel/src/nudge-text.ts ONE_CALL_HINT "ONE call, ONE string" → strict-JSON framing (incidents happen right after a nudge fires)
kernel/src/nudge-text.ts gentle-nudge 💡 line same; "most robust through lossy gateways" deleted
kernel/src/packs.ts lean paramDescriptions.content array-first with escaping rules; string form as quoted value

Unchanged: schema structure (anyOf array|string), the parser (both forms, plus lenient recovery, still accepted), tool names, all other prompts. The host serves kernel text on every surface (wire injection, plugin manifest → MCP, opencode native/V2), so this single change lands everywhere with no override layer — independent acp-kernel consumers get it too.

Golden regeneration rationale

tests/golden/wire-contract/{bili,kernel,manifest-default,manifest-optin}.json — 48 description lines swapped, structure byte-identical, zero occurrences of the old phrasing remain (ONE plain string, no JSON escaping, no JSON structure, lossy gateways — repo-wide grep clean outside this commit message).

Test updates (pinned-wording tests)

  • kernel/tests/compress-schema-contract.test.ts: "stringified-array form documented" → asserts the strict-JSON contract ("strict JSON rule"); header example m00150–m00220 still pinned.
  • kernel/tests/nudge-text.test.ts: static-hint signature "ONE call, ONE string" → "never split the batch across separate calls" (the hint's invariant tail — unambiguous against the gentle line's legitimate "in ONE call —" phrasing).
  • kernel/tests/fixtures/prompt-function-default.txt: TOOLS-section compress line refreshed to the rendered new text.
  • tests/one-call-payload.test.ts: same signature swap; adds strict JSON assertion to the 分析缓存泄露问题又一个日志 #2302 static-hint shape test.

Verification

  • npm --prefix kernel test: 1284/1284.
  • Root: one-call-payload + wire-contract-inventory + search-context-cross-session (feat: 支持 search_context 显式查询历史 session #841 drift guard — BILI vs kernel byte-identity holds without any host override) + compress-tool: 11/11; wire-contract + wire-contract-golden: 21/21; tsc --noEmit clean.
  • Rendered-text spot check: tool description shows the pasteable skeleton with plain quotes; param description shows literal \" / \n escape teaching.

Closes #2587. Evidence: #2579 (stats + character-level attribution), #2580 (independent A-class reproduction). Salvage-side backstop remains orthogonal in #2591.

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-10-10_kernel-strict-json-compress (dd83e91)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.2855-pr.2622.2c16969

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr2622.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Review 结论:方向正确、代码自洽,但当前 head 有 2 个合入前阻塞项——暂不可合。

先对齐编号:GitHub PR 是 #2595(head 36ee132d0,branch 2026-10-10_kernel-compress-strict-json-wording),本条是它的 ework 跟踪 issue #2622;实现的是 #2587。

结构化八问

1. 是什么问题:把 kernel 默认 compress 工具/nudge 文案重写为 strict-JSON 契约(修 #2587——#2579 实测坏 JSON 的提示层诱因本体)。7 处规范面统一改为「参数按 strict JSON 解析,未加引号的值 / 字符串内未转义 " / 换行都会使整个调用失败」,PREFERRED 翻转为对象数组并给可粘贴骨架,删除全部 "no JSON escaping" / "lossy gateways" 措辞。

2. 什么原因导致(两个阻塞项的根因):

3. 影响范围与严重程度:功能不坏。已静态排除确定性文案回归——master 上没有 host override 层(grep 无 COMPRESS_*_FIXED / COMPRESS_JSON_NOTE),kernel 改动干净传到所有下发面;golden 仅 description 行变、结构逐字节一致;旧措辞的钉住断言全部落在本 PR 已更新的文件里,无遗漏的 pinned test;新模板串转义渲染正确(\\"→\"、\\n→\n)。属「合入前必须修」的流程性阻塞,非逻辑回归。严重度:中(照现状合会破坏 kernel 版本纪律 + 带红合入)。

4. 怎么修的(建议动作):① rebase 到当前 master;② kernel/package.json 补到 0.0.113(唯一冲突行,其余文件 master 侧未动);③ rebased head 上跑两层测试(kernel 套件 + bili 套件)+ CI 全绿再合。我已本地重建 merge 态验证:除 kernel/package.json 外,本 PR 触碰的文件在 master 侧零漂移,merge 干净、无 golden 再生成需求。

5. 是否回归问题:逐项过——schema 结构(anyOf array|string)、parseCompressInput 宽容解析器、工具名、nudge cadence 常量、id/ref 不可复用保证均未触碰;gentle 💡 行保留原前缀;line-form 描述保留 Line form 前缀(#2588 patcher 幂等不破)。PREFERRED 由 "ONE plain string" 翻转为对象数组是有意行为变更,已在 PR 正文表格披露(旧值→新值 + 原因齐备)。

6. 是否破坏缓存:机制不变(缓存/重试/游标/去重键均未触碰)。措辞切换对长会话是一次性 system 块改写(前缀缓存一次性失效)——文案修复固有后果,#2588 与本 PR 谁先合谁触发这一次,后合者字节相同不再二次触发。

7. 是否改设计方向:无转向。纯文案层,方向与 #2579/#2588 完全一致(strict-JSON 前置、对象数组优先)。

8. 是否引入新配置:无。无新 env/config/默认值。

判定

暂不可合。 门槛:red 的 CI 必须先转绿。最可能的红因是环境/注册表抖动——master 在本分支切出之后才合了 #2616(ci-npm-registry-retry),本 PR 那次 CI 很可能撞上修复前的注册表抖动;rebase 推新 head 会触发含 #2616 的全新一轮 CI,是判定红因的权威依据,若仍红再定向抓具体失败测试。

我现在就代为执行 ① rebase + ② 版本补 0.0.113(机械、安全、单行冲突),force-push 回 PR 分支触发新 CI,然后按 CI 结果跟进。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 已执行机械修复并推回 PR 分支。 上文八问结论不变(暂不可合→现已解除两个阻塞项的代码侧部分),这里是执行记录:

做了什么

核对结果(vs 当前 master)

剩余门槛(CI)

  • 新 head 已触发一轮全新 CI(此时已含 fix(#2611): retry npm registry fetches in e2e CI setup #2616 的 npm-registry-retry 修复),这是判定上一轮红因的权威依据。我会在 CI 出结果后回贴:全绿即达到可合状态(合并仍由你手动执行——AGENTS.md 禁止 agent 合 PR);若仍有测试红,我会定向抓到具体失败项再诊断,不会拿「大概率是抖动」糊弄过去。

一句话:代码侧两阻塞项已清,现在只等 ba31623a3 的 CI 转绿。

@ranxianglei

Copy link
Copy Markdown
Owner Author

Windows CI triage for acbea29 (and fix c8a4178)

The two red test (windows-latest, 22/24) jobs on acbea29 are pre-existing on the base, not introduced by this PR:

  • Same single test fails: tests/issue2605-dsh-retry-patch.test.ts — AssertionError: compaction-basic auto:false row must stay intact.
  • The base commit 98fb91c fails identically: run 38034641775, jobs test (windows-latest, 22) / test (windows-latest, 24), same assertion at tests/issue2605-dsh-retry-patch.test.ts:27. Last green master (f3efdca) predates the test — it has never been green on Windows.
  • Local run on this branch: 2/2 pass (Linux checkout keeps LF).

Root cause — the #1305 class, exactly as documented in .gitattributes:

  1. The test byte-pins a multi-line needle "- id: compaction-basic\n config:\n auto: false\n" against the repo-checked-out dsh.bundle.patch.yml.
  2. Git-for-Windows' default autocrlf=true CRLF-converts it at checkout on Windows runners → includes() misses.
  3. test: wire-contract anti-regression system — parity fakes, goldens, unified ledger #1305 added text eol=lf for tests/golden/wire-contract/*.json for precisely this reason; the yml was never covered. The runtime-written overlay lane (test 2) is LF by construction and passes everywhere.

Fix: c8a4178 adds one attribute line dsh.bundle.patch.yml text eol=lf (plus comment), following the #1305 precedent. No behavior change on Linux. This unblocks the Windows lane for this PR and for master (#2610's merge broke it; current master head 5b58623 still carries the failure).

Ubuntu + all e2e lanes were green on acbea29; the kernel change itself (strict-JSON compress copy, kernel 0.0.113) has no platform-specific surface.

ework-agent added 2 commits October 10, 2026 17:27
… compress copy (#2587)

The kernel-default compress surface framed the string form as 'plain text …
no JSON structure, no escaping' with UNQUOTED ref-header examples
('m00150–m00220 optional topic'). Models then streamed content values that
are not valid JSON — ~4% of compress calls on strict-JSON hosts (#2579/
#2580): bare unquoted values with real newlines, or unescaped quotes inside
strings — and the host rejects the tool input, failing the WHOLE turn.

Canonical text now states the contract on every surface the model sees:
tool description, content param, line-form item, system-prompt TOOLS
section, ONE_CALL_HINT, the gentle-nudge hint line, and the lean pack:

- strict JSON parsed before anything runs; unquoted values or unescaped
  quote/newline inside any string fail the whole call
- PREFERRED flips to the object-array form with a pasteable skeleton
  ({"content":[{"startId":"m00122",…}]}) — short strings, fixed shape
- the ONE-string form stays accepted but is now described as what it is:
  a quoted JSON string with header lines and markdown inside the quotes
- all 'plain text / no JSON escaping / no JSON structure / lossy
  gateways' phrasing deleted (it taught exactly the broken output)

Schema unchanged (anyOf array|string); parser unchanged; both forms still
accepted. kernel 0.0.112 → 0.0.113. Wire-contract goldens regenerated —
description text only, structure byte-identical. Pinned tests updated to
the new contract assertions; fixture refreshed.
@ranxianglei
ranxianglei force-pushed the 2026-10-10_kernel-strict-json-compress branch from c8a4178 to dd83e91 Compare October 10, 2026 09:27
@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 先纠正一个我的失误:前两轮我把 PR #2595(分支 2026-10-10_kernel-compress-strict-json-wording)当成本 issue 对应的 PR 去 review/rebase 了。核对正文后确认——本 issue 实际包裹的是 GitHub PR #2622(分支 2026-10-10_kernel-strict-json-compress,head c8a41789):正文首行与 #2622 逐字一致、标题完全吻合;而 #2595 是同一 #2587 的另一份并行实现(标题/正文都不同)。我对 #2595 做的 rebase + kernel→0.0.113 本身无破坏,只是对象错了,特此说明。

你的 Windows 诊断我逐条核实过,成立:

  • tests/issue2605-dsh-retry-patch.test.ts 确实对 git-checkout 出来的 dsh.bundle.patch.yml(仓库根)做多行 LF needle 的 includes()("- id: compaction-basic\n config:\n auto: false\n"),断言文案正是 compaction-basic auto:false row must stay intact。
  • #1305 只给 tests/golden/wire-contract/*.json 加了 text eol=lf,没覆盖这个 yml → Git-for-Windows autocrlf=true 把 yml checkout 成 CRLF 后 includes() 必挂;test 2(运行时写 overlay)天生 LF,所以只有 test 1 在 Windows 红。根因判断准确。

但有关键新事实:这个 CRLF 修复已经合进 master 了(在你这条 triage 之前/同期由另一路完成):

另有一个必须处理的问题:#2587 现在有两个 open PR——#2595 与 #2622 是同一改动的两份实现,都基于旧 base 98fb91cc2,现均落后于 master(v0.1.191)。按 loop guard「一个 issue 一个 PR」应二选一。

关键判断(浓缩):

建议(请拍板):

  1. 保留 kernel: teach the strict-JSON serialization contract in the canonical compress copy (#2587) #2622、关闭 [PR] kernel: strict-JSON contract on default compress tool wording (#2587) #2595(kernel: teach the strict-JSON serialization contract in the canonical compress copy (#2587) #2622 才绑定本 issue,且提交链与正文严格对齐)。
  2. 把 kernel: teach the strict-JSON serialization contract in the canonical compress copy (#2587) #2622 rebase 到当前 master(7c993f9ed),rebase 时丢弃冗余的 c8a41789(已被 fix: pin dsh.bundle.patch.yml to LF + normalize CRLF before byte-exact includes (#2631) #2634 完全覆盖且更完善),保留 2579de2aa(kernel→0.0.113)+ acbea29e(first-sight NUDGE_HINT 重钉)。
  3. push 触发新 CI(已含 fix(#2611): retry npm registry fetches in e2e CI setup #2616 registry-retry + fix: pin dsh.bundle.patch.yml to LF + normalize CRLF before byte-exact includes (#2631) #2634 CRLF 修复),预期全绿。

第 2、3 步是纯机械操作,我可立即执行并 force-push 到 …_kernel-strict-json-compress;第 1 步涉及取舍,等你确认我再动。要我现在就 rebase #2622 吗?

@ranxianglei

Copy link
Copy Markdown
Owner Author

Rebased onto master 7c993f9 (head dd83e91): dropped my earlier .gitattributes commit in favor of #2634's identical dsh.bundle.patch.yml text eol=lf fix — same content, their comment. No other changes: kernel/package.json stays 0.0.112→0.0.113 (master hasn't bumped), kernel/src untouched on master since the original base, goldens/repins apply clean.

Local re-verify after rebase: kernel suite 1284/1284; root first-sight-drain-wire + one-call-payload + wire-contract 23/23. CI on dd83e91 all green (main CI + all e2e lanes).

Diff is 13 files +67/−66: semantic change = 7 copy spots in kernel/src (±7 lines); the rest is mechanical — 4 wire-contract golden snapshots (+48/−48, byte-exact so any wording change regenerates them wholesale) + 4 tests re-pinning the old wording + the kernel version bump.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 已独立复核,#2622 现在可合并。 我拉了新 head dd83e9132 逐项核对:分支干净地落在当前 master(7c993f9ed)之上(merge-base == master),冗余的 .gitattributes 提交确已丢弃(diff 不再含该文件),kernel 版本 0.0.113,diff = 13 文件 +67/−66,与你描述一致。CI 我走 API 实查(非转述):dd83e9132 上 26 项 check-run 全绿、0 失败——包括此前红的两个 Windows job(test (windows-latest, 22/24) 现均 PASS,印证 #2634 的 CRLF 修复生效),kernel-guard / version-guard / pi-subagents-guard / docs-sync / lane-guard / build-artifact / 全部 e2e lanes 全过。mergeable=true。

结构化八问(对本 PR 的最终结论):

  1. 是什么问题:内核默认 compress 工具/nudge 文案把 string 形态描述成「纯文本…无 JSON 结构、无需转义」并给未加引号的 ref 头示例,致模型在严格 JSON 宿主流出不合法 JSON(PR 正文称 ~4% compress 调用在 message_stop 被拒、整轮失败)。本 PR 在内核规范副本里明确教学 strict-JSON 序列化契约(kernel 默认 compress 文案仍引导非法 JSON(#2579 的内核侧后续) #2587)。
  2. 什么原因导致:文案本身诱导了非 JSON 输出——根因是措辞而非解析器;修法是「教模型怎么写」,不是放宽校验。
  3. 影响范围与严重程度:所有服务内核 compress 文案的宿主(wire 注入、plugin manifest→MCP、opencode native/V2)都吃到这一处改动、无 override 层;修复前在严格 JSON 宿主上是整轮失败级(高严重度),现已消除。
  4. 怎么修复:改写 7 处规范副本——每个字符串必须加引号、内部 " 写 \"、换行写 \n;PREFERRED 从「ONE plain string」翻转为对象数组并给可粘贴骨架。schema 结构(anyOf array|string)、解析器(含 lenient recovery)、工具名、其余 prompt 全不动;goldens 机械重生成。
  5. 回归问题:内容层上一轮已静态排除确定性回归(无 host override 层、无遗漏 pinned-wording 测试、goldens 仅描述行变、转义渲染正确)。本轮 rebase 只做两件事——丢弃被 fix: pin dsh.bundle.patch.yml to LF + normalize CRLF before byte-exact includes (#2631) #2634 覆盖的冗余 .gitattributes 提交、base 移到当前 master;rebase 干净落上(若 master 动过这 13 个文件会有冲突或 diff 偏差,实际没有)。无 wire/schema/持久化/配置语义变化;高危模式逐项过:fail-fast 未变吞错、无阈值/时序/输出格式漂移(除有意措辞)、id-never-reused 保证未触碰。
  6. 是否破坏缓存:压缩机制与前缀缓存语义不变;措辞改动一次性重写 system/tool 文本块(任何措辞修复固有),不改缓存失效策略;nudge cadence(flat 50K)未动。
  7. 设计方向:不涉及转向——按 owner 方向把契约落进内核规范副本,关闭临时 host-override 路径(fix: serve strict-JSON compress serialization guidance on all host surfaces (#2579) #2588 已关闭未合)。
  8. 新配置:无。

判定:可合并——CI 26/26 独立核实全绿、分支干净 rebase、diff 收敛到「每行都服务于 #2587」。合并动作按规矩留给你(human-only)。

剩一件收尾:重复 PR #2595 仍是 open(同一 #2587 的另一份实现,head …_compress-strict-json-wording)。按 loop guard「一个 issue 一个 PR」,应在 #2622 合并前后关掉它,免得日后误合或困惑。你说一声我就关(或你自己关也行)——要我现在关 #2595 吗?

@ranxianglei

Copy link
Copy Markdown
Owner Author

Closing in favor of a redo with a minimal-diff approach: keep the original ONE-string line-form wording and swap only the JSON-poisoning phrases ('no JSON escaping', 'plain text survives lossy gateways best') for short escape hints — no wholesale rewrite. The strict-JSON root-cause fix (#2587) is preserved; #2591 salvage stays as the backstop.

YunzhouLi-hub pushed a commit to YunzhouLi-hub/billion-context that referenced this pull request Oct 11, 2026
…pe hints (ranxianglei#2587)

Minimal-diff redo of ranxianglei#2622 per review: the ONE-string line-form stays the
PREFERRED shape everywhere; only the phrases that taught models to write
invalid tool-call JSON are replaced, each delta under ten words:

- compress-tools.ts content param: '(plain text survives lossy gateways
  best; …)' -> '(a valid JSON string: internal quotes \" and newlines \n; …)'
- compress-tools.ts line-form entry: 'verbatim (no JSON escaping)' ->
  'verbatim (a JSON string: quotes \", newlines \n)'
- COMPRESS_TOOL description: '(no JSON structure, no escaping; …)' ->
  '(a valid JSON string — internal quotes \", newlines \n; …)'
- TOOLS system-prompt example: '(plain text survives lossy gateways best)'
  -> '(a valid JSON string: \" and \n)'
- nudge ONE_CALL_HINT / gentle 💡 line / lean pack: 'plain string' ->
  'JSON-escaped string'; '(most robust through lossy gateways)' ->
  '(a valid JSON string: \" and \n)'

Wire-contract goldens + prompt-function-default fixture regenerated.
kernel 0.0.112 -> 0.0.113.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

kernel 默认 compress 文案仍引导非法 JSON(#2579 的内核侧后续)

1 participant