Skip to content

feat: chain checkpoint step 3 — per-wire stamp shapes + first-processor-wins enforcement - #1441

Merged
ranxianglei merged 2 commits into
masterfrom
2026-09-26_chain-checkpoint-step3
Sep 27, 2026
Merged

ranxianglei merged 2 commits into
masterfrom
2026-09-26_chain-checkpoint-step3

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Closes #1421

Step 3 of the chain-checkpoint series (#1395 design; step 1 advisory #1364 and step 2 recognition contract #1397 already merged). Implements the owner-approved decision on #1421 (2026-09-26): adopt carrier merge (a) with per-wire split on the role-alternation-strict wires.

Stamp shape per wire (insertCheckpointCarrier)

wire shape rationale
anthropic merge as an EXTRA TRAILING TEXT PART of the last user message (string content normalizes to parts); standalone append only when the body does not end in a user message native Anthropic tolerates same-side runs, but gateways/intermediaries are not all that lenient — never emit them
google merge into the LAST CONTENT'S PARTS when it is not model-role (mandatory) mirrors appendGoogleNudge; the append form broke bili's OWN pipeline before any upstream saw it — coreToGoogle fuses same-side runs into one multi-part content, so an appended standalone user entry no longer matched the recognition contract (pinned as a must-fail regression test through the real kernel function)
openai standalone trailing user MESSAGE (unchanged from step 2) lenient wire; standalone trailing content is legal
responses standalone trailing user message inserted BEFORE any trailing compaction_trigger (unchanged) #283 invariant: compaction_trigger stays the last input item

Restamping an already-stamped body REPLACES the trailing carrier instead of stacking (idempotent restamp).

Recognition relaxation (extractChainCarriers)

Still whole-PART matching (no substring scanning anywhere); ownership semantics unchanged (a client whose final message literally is a well-formed tag binding its own digest still self-opts out — accepted by design):

  • anthropic: whole-content OR a LAST part that entirely is the tag
  • google: ANY part that entirely is the tag (a later hop's nudge merge can push the tag off the end of parts — position is not part of the contract)
  • openai/responses: unchanged
  • strip is surgical: part-level removal for merged carriers, whole-slot for standalone ones; emptied entries dropped

Enforcement (server.ts) — first-processor-wins

verdict behavior
valid forward verbatim, pipeline skipped (info log)
recent-mismatch forward verbatim + warn (interop default)
stale + digest match forward verbatim + warn (replay / clock skew)
stale, no digest match strip the stale carrier(s), process normally → outbound leaves re-stamped fresh by this instance
invalid never trusted; process normally + warn

New ChainCheckpointContext.selectedMatched splits the stale verdict. Kill switch: enforcement AND outbound stamping both respect opts.chainContentDetection !== false (BILI_CHAIN_CONTENT=0).

Outbound stamping (forward())

Every request that actually ran the kernel (prepared with non-empty processedMessages, not side passthrough) leaves with a <bili-chain> stamp computed over the EXACT bytes forwarded — placed after compat roles (#552) and output steering (#1093) so the digest covers them. requestId = first 8 hex of the digest (deterministic, constant-size). Best-effort: a stamp failure logs debug and forwards unstamped. Passthrough / side / forge / classifier forwards carry no stamp.

The module header carries the identification-contract-freeze warning carried forward from #1397: the parser, JCS digest scheme, and verdict matrix are frozen — step 3 changed only WHERE the carrier sits, never WHAT a checkpoint is. Legacy advisory text updated from "until signed chain proof lands" to "until both sides ship the request checkpoint".

⚠ Intended behavior changes (old → new, why)

  1. bili→bili relay with a valid / recent-mismatch / stale-matched checkpoint: previously shadow-log-only + full pipeline ran (double processing) → now forwards byte-identical verbatim, pipeline skipped. This is the point of step 3: first-processor-wins becomes real even when x-bili-hop is stripped in transit.
  2. Every processed outbound request now carries a <bili-chain> tag (~≤200 chars) in the trailing control slot (merged text part on anthropic/google, standalone user message on openai/responses). Rides the tail of the request; prefix-cache neutral by design.
  3. Log prefix [chain-shadow] → [chain]; legacy advisory warning wording changed (see above).
  4. Recognition surface widened: anthropic last-part / google any-part tags are now carriers (previously whole-content only).

Tests

  • New tests/chain-checkpoint-step3.test.ts: per-wire stamp shapes (exact-shape assertions), restamp idempotency, recognition-relaxation matrix, google must-fail regression through the real coreToGoogle fusion (the old append shape breaks recognition inside our own pipeline — pinned, not just against a fake upstream), round-trip invariant across wire × trailing shape, integration E1–E7 (valid → byte-identical verbatim forward; recent-mismatch → verbatim + warn; stale-unmatched → strip + process + fresh restamp; plain → processed + stamped outbound that verifies valid; kill switch BILI_CHAIN_CONTENT=0; side requests leave unstamped).
  • tests/chain-checkpoint-step2.test.ts updated to step-3 semantics (T1/T2 now assert byte-identical verbatim forward; the google two-part case IS now a carrier under the any-part rule).
  • Eight unrelated test rigs pin exact wire shapes and opt out via chainContentDetection: false.

Pre-flight

  • npm run typecheck — clean
  • npm test — 2866 tests: 2854 pass, 0 fail, 12 skipped
  • npm run build — clean
  • request-pipeline hot files touched (src/server.ts) → CI e2e job will run the real-client suite on this PR

@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: 2026-09-26_chain-checkpoint-step3 (b19e366)

Option A — Install this exact build (recommended)

npm install -g billion-context@0.0.1401-pr.1441.ec1b3ac

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Option B — Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr1441.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 开始 review 这个 step-3 PR:先拉分支核对代码与 CI 状态,再按方向关(#1421)/回归关/代码关逐层审。稍后给出完整结论。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Review complete — HOLD: one blocking regression found (both native-lane e2e jobs, root cause proven below). Everything else passes.

CI final state @ 9d381bd4 (all runs completed)

✅ CI unit suite (ubuntu+windows), build-artifact, version-guard, E2E real codex, e2e-fake-codex, e2e-native-dsh, e2e-local-registry, codex overlay (#686)
❌ e2e-native-pi, e2e-native-opencode — both fail model-invoked compress really folds context

① Direction gate — PASS

Matches the #1421 floor-4 owner decision verbatim: per-wire split (google merge mandatory / anthropic merge uniform / openai+responses standalone append), google any-part recognition, round-trip normalization pinned, idempotent restamp, and the must-fail regression through the real coreToGoogle are all present in tests/chain-checkpoint-step3.test.ts.

② Regression audit (existing behaviors touched)

Behavior Verdict
Every processed outbound now carries a ≤200-char carrier disclosed, intended (the feature)
Relay w/ valid checkpoint: double-processed → byte-identical verbatim disclosed, intended
Log prefix [chain-shadow]→[chain], legacy advisory wording disclosed
Recognition widened (anthropic last-part / google any-part) disclosed
Trailing-run scan now breaks at first non-admitted entry (was: skip & continue) not explicitly disclosed — but conservative: worst case is missing a carrier → process normally; never a wrongful skip. Noted, no action
Skipped relays don't bind sessions downstream consistent with #300 hop-marker semantics (pre-existing pattern) ✓
8 unit rigs opted out (chainContentDetection: false) justified churn — they pin pre-stamp exact shapes; each is one labeled line
native-lane e2e rigs (pi/opencode) broken — see blocker

③ Code review

④ Test quality

The new suite has teeth, not tautologies: E1–E7 enforcement matrix through a real rig asserting byte equality of the captured upstream body; round-trip stampOutbound → evaluateChain=valid across 11 base bodies × all 4 wires (string input, compaction_trigger tail, tool_result tail, model tail, multi-part); must-fail twins proving the old google shape loses the carrier through the real kernel fusion. Gap: the stamped default-on wire is not exercised by any lane e2e (see below).

⑤ Diff cleanliness

Single commit, 12 files, every line purpose-related. Clean.


BLOCKER — deterministic breakage of both native lanes, root cause proven

The outbound carrier lands exactly where the lane fake-upstreams key their script/oracle logic — the last user message (part):

  1. pi (anthropic wire, tests/e2e/fake-upstream.mjs:33-40): lastUserText() returns ONLY the last part's text. The merged carrier is that last part → the scripted model sees <bili-chain>… instead of 请调用probe_tool;请调用compress;请调用acp_status → never emits a compress call → 'compress must have been called and its result re-sent' (tests/e2e/e2e-native-pi.test.ts:576).
  2. opencode (openai wire, tests/e2e/fake-upstream-chat.mjs): the oracle's lastUserRef is the ACP-ref regex over the last user message (:179-181). The openai carrier is a standalone user message appended post-pipeline, so it never gets an ACP ref tag → null → 'filler message must carry its ACP ref tag, got null' (tests/e2e/e2e-native-opencode.test.ts:617-620). (The row's lastUser field comes from the first user text, :205 — which is why fillerRow was still found.) The fold run additionally loses its directive: source selection (:133-137) checks the last user msg, falls back to the first (filler, no directives) → no compress call either.

Evidence this PR caused it: both lane workflows were green on their 9 most recent runs before this head — newest wt-1420-ework-daemon @ 2026-09-26T16:06Z, ~18 min before this head's 16:24Z run. Same assertion, ~3s durations, no flake signature: deterministic.

The PR opted out 8 unit rigs but missed these two e2e lanes.

Fix options

  • A (recommended): make the two fixtures carrier-aware — in fake-upstream.mjs, if the last part parses as a whole chain tag, fall back to the previous part; in fake-upstream-chat.mjs, skip a trailing standalone whole-content chain-tag user message when selecting the directive source and computing lastUserRef. Keeps the lanes exercising the shipped default-on config.
  • B (minimal unblock): add BILI_CHAIN_CONTENT: "0" to both lanes' spawn env (kill switch proven by E5). Zero fixture risk, but the default-on stamped wire then has no lane-e2e coverage (unit E-tests partially compensate).

I recommend A; happy to help land it or review the fix commit. Otherwise this is ready — merge once the two lanes are green.


中文摘要:方向关通过(与 #1421 owner 拍板的 per-wire 方案逐条一致)、代码与单测质量合格,但出站 <bili-chain> 盖章把 pi/opencode 两条 native-lane e2e 假上游的脚本逻辑(只认最后一条 user 消息/最后一个 part)确定性打红——根因已定位到具体行,需让这两个 fixture 识别并跳过尾部盖章(A 方案,推荐)或临时以 BILI_CHAIN_CONTENT=0 关闭(B 方案)转绿后再合并。

@ranxianglei

Copy link
Copy Markdown
Owner Author

Deep review (head 9d381bd) — ✅ MERGEABLE, no code changes required

All six axes verified: per-wire stamp shapes match the #1421 owner decisions exactly (anthropic trailing-part merge / openai+responses append / google mandatory merge, compaction_trigger stays last — #283 intact); first-processor-wins decision tree is consistent with #1364 (digest-match outranks fresher mismatched stamps — a foreign fresh stamp cannot steal ownership); restamp replaces in place, never stacks (E5 self-round-trip); malformed/foreign/v≠1 all degrade to process-normally, never break processing; x-bili-hop decisive passthrough fires before the chain block; kill switch gates both enforcement and stamping; stamp never lands on passthrough/side-channel/forge/classifier forwards.

F0 (perf) — measured, acceptable: ~1.2ms @100kb · ~3.1ms @1MB · ~5.6ms @2mb (JCS stringify + SHA-256, linear). A 1M-token session (~4MB) pays ~10ms sync per request — real but small against per-request network+LLM latency (seconds). If it ever matters, a body-size escape hatch is a one-line follow-up; not a blocker.

Needs owner eyes (informational, none blocking)

Gates

typecheck ✓ · chain-checkpoint step2+step3 26/26 · wire-contract+issue1086 31/31 · 4 adjacent opted-out rigs 22/22 — full suite on remote CI. Per AGENTS.md §7.4 this changes wire shape in both compression modes → human merge either way.

Merge order: this lands after #1397 (step 2, already in master) — branch is based on current master tip, no rebase needed.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Agree with the deep-review axes (decision tree, restamp, degradation paths all hold up) — but the MERGEABLE verdict rests on two things that are no longer true at head 9d381bd, both mechanical and small:

1. Two lane jobs are red on this head — "full suite on remote CI" isn't green

e2e-native-pi and e2e-native-opencode completed failure at 9d381bd (my previous comment has the proven root cause: the outbound carrier lands where the lane fake-upstreams key their script/oracle logic — fake-upstream.mjs:38-40 reads only the last part; fake-upstream-chat.mjs:179-181 computes lastUserRef from the last user msg, which the openai carrier replaces untagged).
Fresh datapoint: both lanes went green again today on three other branches (01:36 / 01:46 / 01:49Z) — infra is healthy, the breakage is deterministic to this PR.
Fix per my earlier comment: A (recommended) make the two fixtures carrier-aware (keeps lanes on shipped default-on config), or B set BILI_CHAIN_CONTENT=0 in both spawn envs.

2. Rebase is now required (wasn't yesterday)

Master moved 825f1529 → b371bd04 overnight (#1419 CCR v3 + #1285 price profile merged). Branch is 12 commits behind, and master touched src/server.ts (+27/−1, hunks at old :1329 / :1442 / :1966 / :2176). No textual collision with this PR's hunks (:1240–1288 enforcement, :1458 pipeline condition, :4567 stamp), but per §7.2 the claim of mergeable must be re-proven on the rebased head: rebase → typecheck + full suite + build → CI incl. both lanes green. Your "no rebase needed" was correct when written; it has gone stale.

Informational items — my positions

Suggested sequence

Author rebases onto b371bd04 → lands the lane-fixture fix (A) → CI fully green including both native lanes → human merge (§7.4 mandates human either way — wire shape changes in both modes). No code objections from me beyond that.


中文摘要:深度审查各轴我同意,但「可合并」有两个失效前提——① 本 head 上 pi/opencode 两条 native-lane e2e 是红的(根因已定位,今天其他分支上同两条 lane 已转绿,证明是本 PR 确定性打红而非基建问题);② master 夜间前进了 12 个提交(#1419/#1285)且碰了 src/server.ts,需按 §7.2 rebase 后重跑全套门槛。建议:rebase → 修两个 fixture(推荐 A 方案)→ CI 全绿(含两条 lane)→ 人工合并;F4 的互操作取舍建议先在 #1421 线程里拍板留痕,再让 step 4 在其上构建。

ework-agent and others added 2 commits September 27, 2026 14:02
…or-wins enforcement (#1421)

Owner-approved design (issue #1421, decision 2026-09-26): adopt carrier
merge (a) with per-wire split on role-alternation-strict wires.

Stamp shape per wire (insertCheckpointCarrier):
- anthropic: merge as an extra trailing text part of the last user message
  (string content normalizes to parts); standalone append only when the body
  does not end in a user message.
- google: merge into the last content's parts when it is not model-role
  (mandatory — mirrors appendGoogleNudge; the append form died inside bili's
  own pipeline because coreToGoogle fuses same-side runs).
- openai: standalone trailing user message (unchanged from step 2).
- responses: standalone trailing user message inserted before any trailing
  compaction_trigger (unchanged; #283 invariant preserved).
Restamping an already-stamped body replaces the trailing carrier instead of
stacking (idempotent re-stamp).

Recognition relaxation (extractChainCarriers), matching each wire's emitted
shape — still whole-part matching, ownership semantics unchanged:
- anthropic: whole-content OR a LAST part that entirely is the tag.
- google: ANY part that entirely is the tag (a later hop's nudge merge can
  push the tag off the end of parts).
- openai/responses: unchanged.
Strip is surgical: part-level removal for merged carriers, whole-slot for
standalone ones; emptied entries are dropped.

Enforcement (server.ts, first-processor-wins):
- valid / recent-mismatch / stale+digest-matched → forward verbatim,
  pipeline skipped (log [chain] info/warn).
- stale without digest match → strip the stale carrier(s), process normally
  (the request leaves re-stamped fresh by this instance).
- invalid → never trusted, process normally + warn.
- new ChainCheckpointContext.selectedMatched splits the stale verdict.
- kill switch: both enforcement and outbound stamping respect
  opts.chainContentDetection !== false (BILI_CHAIN_CONTENT=0).

Outbound stamping (forward()): every request that actually ran the kernel
(prepared with non-empty processedMessages, not side passthrough) leaves with
a <bili-chain> stamp computed over the exact bytes forwarded (placed after
compat roles + output steering so the digest covers them). Best-effort: a
stamp failure logs debug and forwards unstamped. requestId = first 8 hex of
the digest (deterministic, constant-size).

New stampOutbound() export; module header now carries the
identification-contract-freeze warning carried forward from #1397 plus the
#1421 per-wire shape decision. Legacy advisory text updated from 'until
signed chain proof lands' to 'until both sides ship the request checkpoint'.

Tests: new tests/chain-checkpoint-step3.test.ts (per-wire stamp shapes,
restamp idempotency, recognition relaxation, google must-fail regression via
real coreToGoogle fusion, round-trip invariant matrix, integration E1-E7
incl. kill switch and side-request no-stamp). Step-2 test file updated to
step-3 semantics (T1/T2 now assert byte-identical verbatim forward; google
two-part case is now a carrier under the any-part rule). Eight unrelated
test rigs pin exact wire shapes and opt out via chainContentDetection:false.
The openai/responses carrier lands as a standalone trailing user
message (#1421 step 3), which masked the real last user message in
fake-upstream-chat's directive-queue source and lastUserRef oracle
(and fake-upstream's ack oracle) — the scripted model-invoked compress
never fired on the pi/opencode native lanes. Filter carrier items from
last-user semantics; conversation content is untouched.
@ranxianglei
ranxianglei force-pushed the 2026-09-26_chain-checkpoint-step3 branch from 9d381bd to b19e366 Compare September 27, 2026 06:09
@ranxianglei

Copy link
Copy Markdown
Owner Author

Rebase + fixture fix landed at b19e366 (option A, per the review):

  1. Rebased onto current master (825f1529 → cae30bb, the 12-commit overnight advance incl. feat: CCR v3 — planning-aware retrieval for search_context (compress.search.planAware) #1419/feat: expose compress.priceProfile {w,r,q} for cache-economics reports #1285's src/server.ts hunks) — clean, no conflicts.
  2. Both fake-upstreams are now chain-carrier-aware: the openai/responses carrier lands as a standalone trailing user message, which masked the real last user message in fake-upstream-chat.mjs's directive-queue source and lastUserRef oracle (and fake-upstream.mjs's ack oracle) — the scripted model-invoked compress never fired. Carrier items are filtered from last-user semantics only; conversation content is untouched, and the lanes keep running the shipped default-on config (no BILI_CHAIN_CONTENT=0 escape hatch).

Local gates on the new head:

  • e2e-native-pi 4/4 (incl. the previously-red model-invoked compress really folds context)
  • e2e-native-opencode 4/4 (same)
  • full suite 2910 pass / 0 fail (an initial 7 golden failures were a stale symlinked acp-kernel 0.0.93 in the worktree; with the branch's actual pin 0.0.95 installed: clean) · typecheck ✓ · build ✓

Remaining before merge is CI green on this head. F4's interop tradeoff (recent-mismatch → verbatim skip for the session's lifetime) still awaits an explicit owner decision in #1421 before step 4 builds on it.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷 Verified at b19e366 — the fix is correct and both lanes are green. Nothing outstanding on my side.

Fix review (b19e3666, test-only, +19/−3 across 2 fixtures)

  • isChainCarrierMsg / isChainCarrierItem match the actual renderer output exactly — renderChainCheckpoint emits <bili-chain v="…" processor="…" issued-at="…" request-id="…" digest="…"/>, so the whole-content regex /^\s*\x3cbili-chain\s[\s\S]*\/\x3e\s*$/ can't false-positive on real prompts and can't miss a real carrier.
  • Filtering is applied precisely where "last user" semantics live (directive-queue source + lastUserRef oracle in fake-upstream-chat.mjs; ack oracle in fake-upstream.mjs) and nowhere else — conversation content untouched, default-on config preserved (no kill-switch escape hatch in the lanes). Exactly option A.
  • Commit structure is clean: feat commit + separately labelled test(e2e) fix commit.

Small correction to my own earlier post, for the record: both native lanes speak openai-completions (e2e-native-pi.test.ts:253-256 points pi at /v1 with api: "openai-completions"), so both red jobs came from fake-upstream-chat.mjs, not from an anthropic last-part path as I attributed for pi. Pi's exact failure mode: the carrier masked the fold prompt, the directive parser fell back to the FIRST user text (the load prompt — which only scripts probe_tool rounds), so compress was never scripted. Same root-cause class, one fixture; the responses-wire fake-upstream.mjs change is sound belt-and-braces for that wire's ack oracle.

State @ b19e3666

Verdict

No code or process objections remain. When the in-progress unit suite lands green, this is ready for human merge (§7.4 mandates human regardless — wire-shape change in both modes). F4's interop tradeoff stays an owner decision in #1421 before step 4 builds on it — noted, non-blocking here.


中文摘要:已在 b19e366 核实——rebase 基线属实、fixture 修复(+19/−3,两个文件)的 carrier 正则与实际渲染器输出精确匹配且只作用于「最后一条 user」语义、两条此前打红的 native lane 在 CI 上均已转绿(主单测套件仍在跑)、GitHub 判定可干净合并;另更正我此前对 pi 失败机制的一处误归属(两条 lane 同为 openai-completions 线,均源于 fake-upstream-chat.mjs 的指令回退被盖章遮蔽)。单测套件转绿后即可人工合并。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[design] chain checkpoint step 3: carrier stamp shape on role-alternation-strict wires (anthropic/google)

1 participant