Skip to content

[PR] integrate: #5901 (snapshot plane) + #6564 (msgid delivery) — pre-merge validation - #6680

Closed
ranxianglei wants to merge 71 commits into
masterfrom
integrate-6564-5901
Closed

ranxianglei wants to merge 71 commits into
masterfrom
integrate-6564-5901

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

integrate: #5901 (snapshot plane) + #6564 (msgid delivery) on master

DRAFT — validation vehicle, not a merge target. This branch merges the two
open PRs (#5901 + #6564) onto the latest master to run the full remote CI +
local gate:local
against the combined tree, BEFORE either is merged. It
proves the two land together cleanly and green; it is not itself what gets
merged (merge = owner, per repo rule).

What this is

Local validation on this branch

Why a draft, why not merge this

Two open PRs each need their own owner merge (repo rule: no non-human merges).
This branch only de-risks the combined rollout: if both land green together
here, the owner can merge them in order (#6564 → #5901, or #5901 → #6564 — this
branch proves order-independence of the tree).

ranxianglei and others added 30 commits November 14, 2026 15:38
…6539)

Complements the client-side envelope wiring (#6532 shipped dsh openai+
anthropic). This is the 'record' half of the msgid lifecycle made whole
across every wire:

- prepare-google / prepare-responses: read + strip the biliMsgIds envelope
  before it can reach the upstream, then pin behind the same rollout gate
  as the openai/anthropic twins. A length/shape mismatch is rejected
  (bail) at the pin — never leaked, never mis-pinned.
- handle.ts: a non-body id source (x-bili-msg-ids header / loopback
  harvest) now stamps the parsed body on EVERY wire, not just
  openai/anthropic, so the four pin-consumers all see it uniformly.
- rollout: pluginLaneServerHistory 0% -> 100% (GA, owner ruling
  2026-11-13). No client calls the co-gated /__bili/plugin/render
  endpoint yet, so flipping only activates pinning. Roll back with
  BILI_ROLLOUT_PLUGIN_LANE_SERVER_HISTORY=0.

Three tests that pinned the old default-off contract now force the off
state explicitly (env=0) instead of deleting the var.
Principle 1 of the four-principles proposal (discussion #6262): identity,
not bytes, decides which prefix a fork inherits. The plugin-protocol
channel (dsh/omp/pi declared forks) gains an optional msgid track:

- a host-pinned parent exposes the host's stable message ids AS its
  snapshot rawIds, so the child may present the host id of each matched
  prefix message (prefixMsgIds) plus the hash-run length (driftFrom);
- the server accepts a byte-drifted prefix only when every presented id
  equals the parent's rawId at its slot; the receipt is marked
  status "drifted" (or "expanded+drifted") with the verified
  driftFrom echoed;
- a gate failure never fails the fork: the byte-verified strict fork
  proceeds exactly as before (no 409); malformed shapes are 400; parents
  without host-pinned ids make the track a no-op (byte path unchanged).

Client: lanes inject their existing parity-pinned collectors (dsh
hostMsgIdsOf; pi/omp laneMsgIdsForWire on the anthropic wire with the
same content-pinnable gate the incremental pin uses); fork-adopt
performs only the core-alignment length lock — no new validator copies,
msgid-rule-parity matrix untouched. opencode-v2 is not wired in this PR
(byte track, follow-up).

Tests: tests/issue6262-fork-msgid-track.test.ts (8) — client prefix
extension across a drift cell, the byte-fallback arms, a full-chain
proxy round trip, and the server gate (drifted receipt / strict
fallback / 400 arms). Docs: msgid-infrastructure.md consumer 6,
plugin-protocol.md section 8, architecture.md.
Carrier ① (the envelope body field) had zero client wiring: the server
consumes biliMsgIds on both wires (prepare-openai/anthropic, gated on
pluginMode + the pluginLaneServerHistory rollout), but no client ever
stamped it. dsh's stamping lived only on the anthropic-wire incremental
lane (#4491), so every openai-wire full-resend turn — dsh's main path —
arrived id-less and the store fell back to content-derived digests,
which in turn leaves the fork msgid identity track (#6293) with nothing
to match.

maybeStampEnvelopeMsgIds now stamps full-resend model bodies (both
wires) after the incremental lane had its chance. Resolution never
re-implements the host's wire transforms: the pi-ai lockstep walkers
(laneMsgIdsForWire / laneMsgIdsForOpenAiWire — the SAME functions the
pi/omp lanes pin with) prove wire-element ⇔ host-message pairing by
order-preserving fingerprints, bail all-or-nothing; the kernel's own
converters then prove core-alignment via the #3642 wireIndex
back-pointer (every non-leading wire element must own exactly one core
— parallel tool calls / reasoning+text / text+image parts split one
host message into N cores, and N cores cannot share one host id).
System-role entries are excluded (they ride the hoisted/folded leading
system, never 1:1 to a non-leading wire element). Any bail leaves the
body byte-identical → content-derived ids, exactly today's behavior;
with the rollout off the server strips the field, so default installs
are untouched.
#6525 B7)

dsh's flat host messages differ from the pi-ai shape the shared walker's
host-side fingerprint assumes in two places (role tool vs toolResult,
block tool-call vs toolCall), so every tool-bearing turn bailed the
all-or-nothing envelope and the turn-1 host-pinned rows churned back to
content digests on the next full-resend. Normalize dsh's shape for
fingerprinting only, in the dsh resolver: the envelope ids still come from
mm.id and the wire-side pairing is untouched, so no mis-pin is possible.

Found via a real-dsh e2e dump (after turn 1: live rows 11/11 content
digests). Now: live rows 11/11 host-pinned UUIDs. Two regression tests pin
the accept (a dsh tool turn core-aligns and stamps) and the retained bail
(parallel tool_calls still refuse, one host id cannot cover N cores).
…6525/#6539)

The server pins host ids against the POST-HOIST core count (prepare-openai
validates ids.length against openaiToCore(parsed).length, which hoists the
contiguous leading system/developer run out of the fold space). opencode V2's
x-bili-msg-ids header was stamped per WIRE message, INCLUDING the leading
system, so its count always exceeded the core count on the openai wire and the
pin was rejected (a safe content-derived degrade — but opencode V2 never
actually pinned on the openai wire).

Core-align the V2 extractor: skip the contiguous leading system/developer run
before collecting ids so the header count equals the post-hoist core count.
No-op on the anthropic wire (system is a separate field, never in messages);
mid-conversation system messages keep their ids; a leading-system-only turn
degrades (no empty header). Whole-turn poison contract unchanged.
…#6539 review)

Dual-agent review of the #6539 integration found the egress gap: a stamped
full-resend body rode biliMsgIds/biliLedgerSeqs out through the verbatim
forward branches that bypass the four fully-processed prepares — strict-schema
upstreams 400 'Extra inputs are not permitted', and #661 body fingerprinting
breaks.

- server.ts: protocol-neutral scrubMsgIdFields (byte-identical pass-through
  when the fields are absent, so #661's contract is untouched in the normal
  case) + prepareCountTokens egress hygiene.
- handle.ts: scrub applied on every verbatim branch (unattributed passthrough,
  bypass, v1internal background, commandcode unconvertible, no-messages
  fallback, side-request body, auto-mode forge, final fallback). The #1884
  signed-passthrough escape hatch is deliberately NOT scrubbed — the
  body-covering sdk-hmac-sha256 signature would break; byte-untouched wins.
- dsh-native.ts: resolveEnvelopeMsgIds now calls the shared
  coreAlignEnvelopeIds instead of the cherry-pick'd inline copy (review M1);
  the fork-adopt msgids use the SAME core-aligned resolver (m1: the old
  per-wire-message hostMsgIdsOf could never pass the adopter's
  msgIds.length===core.length lock); explicit incremental-lane guard (m2);
  ENVELOPE_MSG_IDS_FIELD hoisted to module level with a parity unit test (n2).
- opencode-v2-msgids.ts: comment corrected to the actual responsesToCore
  sysDevHead / anthropic separate-field rules (m3).
- non-ASCII note text removed (n1).

New: tests/issue6539-verbatim-scrub.test.ts (7).
Regression: msgid/envelope/fork unit 70/70 + 28/28 + 22/22; wire e2e
google+responses 17/17; plugin-fork 52/57 (5 pre-existing cold-restore fails).
)

opencode 1.x's ONLY host-id seam is experimental.chat.messages.transform
(harvested per-session by harvestMsgIdsTransform); the wire body only exists
at fetch time. So the alignment runs at the shared fetch chokepoint via a
new state.beforeSend — proving the harvested turn POSITIONALLY against the
ACTUAL wire (alignedMsgIdsForWire), never a verbatim copy of the stored
list: a same-session A-transform/B-transform/A-fetch interleave degrades to
content-derived instead of mis-pinning.

- anthropic-wire only; a turn pins only when every message lowers 1:1 to one
  core (pure-text user/assistant) — tool/reasoning parts or any count/role/
  text drift drop the whole list. The server re-proves count parity +
  canonical shape and strips the field before any upstream forward (#2821).
- Also feeds the shared watermark slot (pfCaptureIds) with the SAME proven
  list — this replaces the DEAD call that sat in harvestMsgIdsTransform
  (it fed host objects {info,parts} to a wire-shape proof, so it could only
  ever clear the slot; V1 watermark was count-only with no tail-ids).
- Guards: never restamp an incremental-lane (diverted) turn, never clobber an
  already-present x-bili-msg-ids (a V2 request stamps its own), budgets match
  the server parser (4096 ids / 16KiB).

New: tests/opencode-v1-envelope.test.ts (9).
…6539)

Boots a real startServer + fake upstream (no agent) and drives the exact
envelope the dsh/pi/omp/opencode clients stamp:
  A. normal plugin lane — biliMsgIds pins every host id into the store (the
     plugin snapshot's rawIds come back as the host ids, not content digests)
     and the field never reaches the upstream.
  B. verbatim (bypass) lane — the same stamped body is scrubbed before the
     forward (the F1 review fix), so biliMsgIds never reaches the upstream.

2/2 pass; typecheck clean.
The dsh/pi/omp full-resend envelope (biliMsgIds) only covered the openai and
anthropic wires; a /responses request (body.input) safely degraded to
content-derived ids. This adds the responses dialect so all three host agents
carry host msgids on every wire the proxy can pin.

- pi-msgids.ts: laneMsgIdsForResponsesWire + respWireFingerprint/respUserParts
  (lockstep pairing via the shared makeEntryTake/partsLowerThrough; conservative
  bail on mid-history system, opaque items, reasoning, and any item the kernel
  would split into N cores). coreAlignEnvelopeIds now accepts "responses" and
  proves 1:1 against the kernel's responsesToCore wireIndex — the perEl check is
  the final no-mis-pin safety net.
- dsh-native.ts / pi.ts: resolvers accept the responses wire (wire = parsed.input).
- tests: 11 new cases (1:1 tool round, lead-system hoist, multi-part/mid-system/
  opaque/reasoning bails, dup-id bail, resolver pairing with and without lead).

Regressions green: pi-omp 34/34, dsh 15/15, opencode-v2 10/10, fork-track 8/8,
msgid full suite, wire e2e google 4/4 + responses 13/13.
… write)

Implements the snapshot-table compression foundation from discussion
snap_rows / snap_log / snap_ckpt, schema v12), the single-transaction
commit primitive with S4 checksum verification, and the P1 shadow
write that derives the plane from final DB state after each ingest.

P0 (zero behavior change, flag default OFF):
- schema v12: the four plane tables + idx_l1_ledger, created by
  migrateSchema (from < 12) and fresh installs; purely additive,
  rollback = DROP the four tables + user_version rewind (T1 pins the
  byte-identical legacy dump in both directions).
- src/storage/snap.ts: the commit primitive (applySnapCommit /
  applySnapInTransaction) — events apply in order, one snap_log entry
  per event with the post-apply S4 aggregate, seq strictly monotonic
  past max(log max, ckpt_seq), optional fold into snap_ckpt (three-
  piece meta: kstate verbatim + coverage mapping + S4 aggregate, log
  truncated at the fold). Read-side rebuild (readSnapView) recomputes
  the aggregate and kills the plane for the sid on any mismatch (R1,
  sticky). L1 identity ledger (snapL1Record): issue/arm/echo/hit/wait,
  hard-refuse on a raw-id → ledger conflict (R3). GC releases rows
  only after their cover vanishes (invariant iv).
- two-level flags: storage.snapBase.enabled (master) ⊃
  storage.snapBase.shadowWrite (lane), config-file only (#2030),
  resolved in knobs (fileSnapBase) and re-asserted at the store.
- /__bili/status gains a snaplane field (flag state + counters + dead
  latches); sessionStorageStats gains the plane dimensions.

P1 (shadow write, read side still never serves):
- after each committed ingest, one store transaction records the L1
  decision and applies the derived events: steady-state delta
  appends take a fast path (no full scan); everything else is a
  final-state diff over live rows + block coverage (cast/update/
  delete/compact, pos-ordered, deterministic). A round that cannot be
  addressed produces no events (R2); a crashed gap self-heals on the
  next save because the derivation is a final-state diff.
- fold at 128 log entries or on any compact; the folded plane is
  checked against the legacy view (A/B oracle) — a mismatch kills the
  plane for the sid. Covered positions are legitimately inactive in
  the plane and are accounted for in the ckpt coverage map.
- deleteSession cascades the four tables (no orphans, T6).

Tests: tests/issue5874-snapbase-p0.test.ts (T1/T1b/T2/T2b/T2c/T2d/
T6 + flag nesting) and tests/issue5874-snaplane-p1.test.ts (T3 B1
trap nail / T4 A/B incl. crash gap / T7 coverage map / T9 bounded
growth). Schema-head assertions in the existing migration suites
bump 11 → 12.

Timing dependency (#5855): the unmerged fastpath re-land also numbers
its two table groups v12/v13; whichever lands second renumbers its
block mechanically (independent CREATE IF NOT EXISTS shapes).

mysql backend: the plane is sqlite-only for now (the async backend
opens without the flags).

Refs discussion #5874.
…5874)

snapGcCoveredRows removed rows outside the commit primitive without
updating the anchor they aggregate over: after an ordinary host-side
rewind (raw-tail historian sweep retires rows, the diff flips them
active=0 with cover_ref='deleted'), the immediate release left the last
log sumAfter covering rows that no longer exist, so the next
readSnapView tripped R1 on a healthy plane and the sticky dead latch
kept the session's plane closed until process restart.

Removal now re-anchors in the same transaction (a pure-reanchoring fold
at the current log tail), preserving the intended release timing.

Also adds the missing P1 fast-path coverage: every prior P1 test saved
through planIngest(full), so snapFastAppendEvents never ran in CI. The
new suite drives the real watermark/delta machinery (UnifiedShadow.record):
steady appends ride the fast path ('append' kinds), and a rewound tail
voids the delta, converges byte-for-byte through the diff, and keeps the
plane alive across the release round (regression pin for the R1 latch).
The derivation reads final DB state, so how the legacy rows arrived
(complete log vs delta) is irrelevant to it; the param was threaded but
never read.
Temporary soak change (revert before merge per owner instruction):
when the config file is silent, the snapshot-table plane now defaults
to enabled + shadowWrite (was off). Every hermetic test and the local
real-traffic gate therefore runs the shadow write path on all
sessions; the config file can still pin it off explicitly.

Spots to revert (all marked 'SOAK' in comments):
- src/storage/store.ts: field default + open() fallback
- src/server/admin.ts: snapplane status default
- tests/issue5874-snapbase-p0.test.ts: default-expectation asserts
- tests/issue5874-snaplane-p1.test.ts: crash-gap store pins OFF
Per-table storage relationship map for the snapshot-table foundation (#5874,
this PR's schema v12): every table card shows what it stores, when it is
written, who reads it, and which feature it serves — ingest, shadow-write,
fold, egress (legacy today / plane in P2), hydration, deleteSession
cascade, GC, and the /__bili/status surface. Registered in docs index.
…ries

The design doc pair (proposals/snapshot-compression-foundation) moves from the
proposal branch into the implementing PR, per the docs-centralization standard:
- bilingual README.md / README.zh-CN.md + 4 arch diagram pairs (svg+png)
  + the arch-migration generator script
- status line updated: P0+P1 implemented in #5901 (schema v12, shadow-write
  soak, zero egress change); P2 egress cutover still pending owner ruling
- §5 version-stacking corrected to reality (landed v12 on v11 master while
  #5855 is still open; later-landing side renumbers mechanically)
- §9 keys finalized (storage.snapBase.enabled / .shadowWrite, config-file only)
- §12/§13 notes: P0+P1 shipped as one PR; #5855 prerequisite not waited on
- cross-links to the per-table map docs/storage-relations.{en,zh}.svg
- registered in docs/index.md + docs/index.zh-CN.md
…th (#5874)

docs-sync red on fb17874: the committed generated rows claimed default
'on (soak; off in release)' while the canonical seeds
(website/config-reference/process.yaml) say default false — the generated
artifact had drifted from its source. Rerun tools/gen-config-docs.mjs
generate and commit its output (2 rows per file, en+zh).

Note for reviewers: the #5901 soak (dab9292) makes the code default ON
when the config file is silent until the owner-instructed pre-merge revert;
this doc describes the seed state, which equals the post-revert code.
…hema versions

Field incident (2026-11-08, shared index.db): the unmerged #5855 re-land
consumed schema v12+v13 (pf_* planes) into a database this branch's
v12 block had never seen. migrateSchema's early return
(from >= STORAGE_SCHEMA_VERSION) skipped the DDL, so the first save
died with 'no such table: l1_world' and every session's plane was
marked dead (R2, by design — but the root cause was the version gate).

The DDL is pure CREATE ... IF NOT EXISTS (no backfill, no rewrite), so
it is safe at any from-version: ensure the four tables by
TABLE-EXISTENCE PROBE before the early return (the discipline v2/v5/v6/v8
already use), instead of gating on user_version < 12. Both merge
orderings now coexist; the final master version number stays a
mechanical merge-time hygiene step. user_version is never downgraded
(the early return leaves it alone).

T1c pins the exact field state: v13 database with sibling pf_* tables
and no snap tables → open → snap tables created, version stays 13,
sibling rows untouched, legacy dump byte-identical, plane functional.
14/14 suite green, 35 storage regressions green, tsc clean.
…CI)

readPkgVersion is polled every 500ms by waitFor in the registry/advisory e2e suites; a poll landing inside the updater's remove-and-reinstall window hits a transiently absent package.json and the raw readFile throws ENOENT out of the predicate, killing the subtest at ~1.6s despite a 60s budget (ubuntu red job 128697545236; windows green on identical code = race). All four copies of the helper now return null on ENOENT ('not yet', not an error); one-shot assert sites still fail loudly on absence.
…-for-byte guard (#5874 P2)

The outbound compressed context is now assembled from the verified
snapshot plane (mapEgress) instead of the in-memory history, on all four
protocol lanes, guarded by a tripwire: the plane view is built first
(anchor summaries at each covered region + surviving rows + the client
tail, the same seam walk as the inbound view), the lane's exact
post-processing is reapplied, and ONLY a normalized exact match switches
the outbound bytes. Any mismatch keeps the legacy bytes; after three
mismatches the session is sticky-excluded for the process lifetime.
Every degrade reason surfaces on /__bili/status under snaplane.egress.

- new src/server/snap-egress.ts; wired into prepare-{openai,anthropic,
  google,responses}.ts right after the legacy processedMessages
- store accessors: UnifiedStore/UnifiedShadow/SessionStore.snapEgressView
- config: storage.snapBase.egress / egressCohorts (soak default ON at
  100%, release cut flips to opt-in)
- admin status wire: snaplane.egress {cutover,fallback,planeMiss,
  excludedSids,degradeReasons}
- tests: 16 new cases (view assembly incl. reply-lag/rewind/remint/gap
  seams, tripwire normalization, anchor rendering parity, gates + sticky
  exclusion); docs en/zh updated (configuration, proposal record, index)
…ed prefix rows

The client harness wraps every resent message in a per-RESEND envelope tag
(<acp tokens=.. time=..>mNNNNN</acp>) whose time attribute changes each turn,
so the snapshot plane can never store it byte-for-byte. That made the
tripwire fire on every folded dsh turn (shape@0:text) even though the plane
content was identical, and sticky-excluded those lanes from the cutover.

The egress view now pairs each surviving plane prefix row with the identity-
matched message from THIS resend (id first, then envelope-stripped signature)
and adopts its exact tagged text, but only when the pair is content-equal
modulo the envelope — otherwise the plane text stands and the byte-strict
tripwire judges it. Envelope changes between turns still seam-match because
both bySig indexing and the seam walk compare envelope-stripped signatures.
…, lock sync, internal types

- website/config-reference/process.yaml: seed storage.snapBase.egress /
  storage.snapBase.egressCohorts and regenerate docs/configuration*.md
  (config-surface-guard, docs-sync)
- reference/architecture.md: register src/server/snap-egress.ts
  (architecture-docs)
- package-lock.json: sync the kernel workspace to 0.0.189 (lock-sync)
- snap-egress: drop export on the file-local SnapEgressDeps /
  SnapEgressStatusWire interfaces (unused-exports)
…-refs pass over the plane view

The <acp tokens type time>mNNNNN</acp> tags are NOT client resend data —
they are system metadata rendered by the kernel's render-refs node (the
LAST pipeline node) over turn.messages. The earlier per-resend envelope
adoption was therefore the wrong model. The plane view now reaches byte
parity by re-running the kernel's own renderer (renderVisibleRefs with
the deterministic default counter) over the assembled view, using the
same post-turn state the lane already holds: refs from
state.messageRefs, immutable first-sight times hydrated from
state.tsStamps, strategy from the lane's renderTags knob, timestamp
mode from resolveMessageTimestamps (env > file > default ON). Render
throws degrade honestly (render-error, legacy bytes kept).

Lanes pass renderTags; T18a/b rewritten to assert kernel-render parity
and first-sight time stability; T18c retitled to the content-edit
divergence case. 19/19 + regression suites green.
ranxianglei and others added 14 commits November 14, 2026 16:10
…-body M2 retries no longer wipe the plane, and the first chained round seeds its frontier at the stored-world boundary (gated on a trusted, never-refused chain)

- Residual 1a: an empty delta (frontier already covers the world — the
  host retried the same body after an upstream failure) is now classified
  as a NOOP: the chain id is kept, no developer message is injected, and
  the empty input rides the provider's own cache. Before: id stripped +
  dev injected = a blind response the host would chain from forever.
- Residual 1b: stage 0 recognizes a full replay (every incoming id already
  in the stored canon, carrier seqs at/below the plane frontier) as
  echo+replay — zero side effects. Before: an M2 retry ([m5] vs head m1)
  read as headMoved -> diverge -> snapStage0Reset: a harmless retry wiped
  the plane.
- Residual 2: the first CHAINED round of a keepResponseId chain carried no
  committed frontier (round 1 had no id), so it re-seeded the whole world
  the provider's chain already holds from this proxy's own round-1 full
  view (2x the world's tokens, one time). Stage 0 now proves the safe case
  (probe + nothing gone + new elements -> the chain provably holds the
  stored world) and seeds the frontier at the stored-world boundary;
  everything else keeps the plain re-seed.
- B4 hardening: a REFUSED selection round now sets a persisted
  responsesChainBroken marker (committed with the session). While set, the
  frontier seed is blocked and the round re-seeds the full view — the
  in-request flush wrote the refused turn's rows before the forward, so
  the plane holds more than the provider's chain and trusting
  storedFrontierPos would lose the refused content. The flag clears on a
  full-view/noop acceptance; the pending delta now carries its kind
  (full/noop/partial) so the relay can maintain it.
- Tests: p3 42/42 (new: residual-2 unit shape pin, residual 1a+2 e2e
  through the proxy, A1/A1d legs now use isolated DBs — the two legs are
  ALTERNATIVE client behaviors on the same turn and sharing a file let
  leg M1's canon contaminate leg M2's verdict).
- Regression: p0 10, p1 4, p2 17, fastpath 2, snapshot-render 13+6,
  snapshot-shadow 5, storage-engine-proofs 12, opencode-rolling 10,
  responses-previous-response-id 7 — all green.
…a delta frontier only for an id this proxy minted (#6181)

previous_response_id proves nothing about which world the provider's
chain holds. A host forking/rewinding to an older response id (or
carrying an id another proxy minted) can pass the strict-extension
proof yet lose the gap between the old chain's world and the stored
world when we seed at the stored boundary.

- session.lastMintedResponse {id, frontierPos}: the latest response
  id THIS proxy emitted for the session, paired with the plane frontier
  the chain provably held — recorded in the relay from the id observed
  on the outbound stream (SSE response.created/completed scan on the
  exact bytes written to the client, json.id for non-streaming),
  accepted forwards only (a refused round mints nothing the client
  chains from).
- store facade snapPlaneMaxPos (store/shadow/persist) — the lane
  records the post-flush plane max pos as prepared.mintFrontierPos.
- seed gate: the first-chained-round frontier seed now also requires
  lastMintedResponse.id === previous_response_id AND
  lastMintedResponse.frontierPos === storedFrontierPos.
- persisted-frontier trust: a chained round whose id this proxy did not
  mint last drops the persisted frontier and re-seeds the full view
  (id kept, dev injected — the pre-seed status quo, at-least-once,
  never a loss); the persisted state is cleared so the next round
  re-establishes verifiability from a fresh seed.
- B3 latent bug: the pure-chain-continuation route reuses the
  compact-passthrough base, which hard-codes resetAfterSuccess —
  scheduling a native-compaction rebase (resetSessionCompression) for
  the next turn although nothing compacted. Dropped for the B3 route.
- p3: per-round upstream ids in the e2e harness; the Residual 1a+2
  e2e chains from the ids the responses actually carried and gains a
  round-4 fork case (old id + full current world + new element →
  full re-seed, id kept). 41/41.
…ate)

The Residual-2 hardening commit added a raw mintSseTail.slice(-32768)
which the text-safe family gate (tests/text-safe.test.ts) rejects — raw
negative string slices reintroduce the #816/#1615 lone-surrogate family.
Replace with safeSuffix(mintSseTail, 32768) (same last-32768-code-units
semantics, surrogate-pair-safe). safeSuffix was already imported.
… rule) + mint tap spin

Two regressions surfaced when the branch was re-based onto the #5901 head
(f000197, which carries #6326 durability-before-render):

1. B4 e2e: #6326's ingestRowsNow persists the ledger rows BEFORE render as
   an UNTAGGED save; its final-state diff then deleted/re-homed the pinned
   plane rows (ids the tagged plane cannot address — R2 doctrine) and the
   next tagged save marked the plane sticky-dead. snapShadowSync now
   guards untagged saves against a sid that holds active plane rows with a
   PREFIX RULE: the save is allowed only when its ledger-id view
   prefix-identically extends the stage-0 l1 envelope (that save is the
   one that materializes the rows stage-0 declared — ids agree,
   idempotent); a stale view (re-derived kernel ids, pre-turn transcript,
   compaction) is skipped for the plane, the disk record still lands, and
   the plane converges at the next tagged save or stage-0 resync. The
   guard runs BEFORE the transaction (after it the destructive diff is
   already committed).

2. Residual-2 e2e: the mint-id scan (the record that lets the next chained
   round seed the delta frontier at the stored-world boundary) existed
   only in the compress-loop (injected) branch; plugin and non-injected
   responses turns never entered it, so the first chained round re-seeded
   the whole world. The scan is factored into MintIdTap and rides a tee'd
   twin of the plugin-lane stream and of the plain (non-rewrite) proxy
   stream — both branches where the client-facing bytes ARE the upstream
   bytes. The tap fix also repairs a latent infinite-spin in the old
   inline scan: the event-boundary recompute ran only at the loop bottom,
   which the non-marker  skipped — a non-marker event preceding
   a marker event pinned the boundary on a shrinking tail and spun the
   loop forever (event loop blocked; no timers fired). The recompute is
   now unconditional on every pass.

p3 42/42 (incl. B4 frontier-commit-on-accept + Residual 1a+2); p0 10/10,
p1 4/4, fastpath 2/2, p2 17/17, responses-previous-response-id 7/7,
snapshot-render 13/13, render-chain 6/6, shadow 5/5, storage-engine-proofs
12/12 (kernel rebuild for #5901's compress.ts), opencode-rolling 10/10.
…wline in the #5901 merge

The 941447a merge joined the egress-cohort desc_ja value with the next
'- key: sessions' entry onto one line, which tools/gen-config-docs.mjs
rejects (quoted value not valid JSON) — docs-sync + docs-coverage red.
Split the line back; generator checks pass (coverage ok: 204 env + 63 keys;
i18n ok: 254 keys).
… v11 test title, comment indent

- pr-body-6299.md: #6299's PR-body draft (already merged) was committed at
  the repo root — authoring artifact, not repo content (violates the docs
  layout: root keeps README*/AGENTS/CHANGELOG only)
- tests/storage-ingest-watermark.test.ts:594: title said 'migrates to v11'
  while the body has asserted v12 since this PR bumped STORAGE_SCHEMA_VERSION
- src/server/prepare-openai.ts:204: comment block started at column 0
…6480's pure-sqlite default flipped the absent-selection JSON writer off, and these suites assert the legacy JSON persistence surface (session files / cold-disk-restore content-store reads). First exposure: the ci:full gate ran them on the #6480-merged tree; reproduced on bare master, fixed by the explicit dual-write pin.

#6480
… missed by the #6480 pin commit (the ci-e2e-dsh-native workflow runs TWO suites; persona asserts raw-conversation-key session persistence too).

#6480
… the CI storage pin in notice-registry (master #6480 regressions)

Two master-side regressions first exposed by this branch's full test shards:

1. fork-ccr-adoption reads the companion session ENVELOPE off disk, but the
   #6480 pure-sqlite default stops writing JSON envelopes. Pin dual-write
   in the test (same one-liner as the native e2e suites).

2. The notice-registry suite is hermetic by design (absent config file,
   fresh XDG) but the CI test baseline pins BILI_STORAGE_UNIFIED=legacy
   (preload-storage-baseline.mjs, #4011). An explicit selection is a user
   decision that legitimately hides the fresh-install default-flip notice,
   so the two 'fresh install sees the default flip' assertions failed on
   every shard. Lift the pin for this file only.

Master merged #6480 without a main-CI run (the push event was dropped), so
neither regression had been seen in CI before this branch.
… pin (Windows EBUSY/EPERM)

persist-metadata-fold-gate.test.ts to keep the legacy envelopes their
assertions read. unified=full opens index.db, and Windows cannot unlink a
file whose SQLite handle is still open — both suites rm'd their temp dirs
without closing the stores, so the windows-latest shards failed
deterministically (EBUSY unlink index.db / EPERM rmSync tmpdir) while every
ubuntu shard passed. Close the stores (closeQuietly) before rmrf.

Same master-side regression class as the notice/pin fixes in 7b6eb98:
master merged #6480 without a main-CI run, so none of these had been seen.
…t-off revert, test coverage, docs

- B0: on non-cohort chained bypass rounds the in-request flush was the only
  consumer guarded by the one-shot skipPlaneFlush; the relay success path
  re-arms the debounced save and that save rode turn.messages' DELTA view as
  a tagged world — snapViewEvents (final-state diff) deleted every plane row
  the delta lacks. The lane now clears session.lastPlaneMessages on bypass
  rounds, so later saves derive untagged from the DB final state (append-only
  rows — correct).
- Default off (discussion #5874 section 9): the 2026-11-08 100% soak default
  is reverted in all four spots — store.ts field + open() fallback, admin.ts
  status payload, snap-egress.ts gate (egress now opt-in, v?.egress === true) —
  and the p0 flag test now pins default-off. e2e suites arm the flags
  explicitly via a dedicated config file.
- Test gaps from the review: T2 covers the in-transaction rollback leg
  (valid append + R2-unknown update in one batch => nothing lands); T9 raises
  N 40->160 so SNAP_FOLD_DEPTH (128) actually fires and asserts a fold
  happened (ckpt_seq >= 1) — at 40 saves the bound was vacuous; the
  byte-identity oracle's LEGACY_TABLES now includes kv_store (was 11 of 12);
  p2 G2-G5 set egress: true explicitly (they had relied on the soak default).
- Docs: process.yaml egress default false + 3-lang regeneration; proposal
  READMEs (EN/zh) status line, DDL block (actual v12 columns incl. envelope /
  r3_latched_at), probe-first migration note, stage-5 row, section 9 key list,
  section 10 metrics all updated to the direct-authority + default-off state;
  config.ts schema comment; stale comments fixed (session.ts lastPlaneMessages,
  handle.ts B3 string-input admission, snap.ts l1 upsert, p0 T6).
The #5901 rebase auto-merge kept BOTH ../persist.js imports: master's
standalone 'import { getStore }' (added for the #3397 ingestRowsNow
durability-before-render call) and this branch's 'import { SessionStore,
getStore }' (added for the in-request SessionStore.flushOrSnap plane path).
Both are needed symbols (getStore at the ingestRowsNow + budget calls,
SessionStore at the flushOrSkip call), so the superset import stays and the
redundant standalone getStore line is dropped. Fixes TS2300 duplicate
identifier surfaced by the post-rebase typecheck.
…seeds (docs-sync guard)

The #5901 rebase resolution of website/config-reference/process.yaml dropped the
desc_ja lines for storage.snapBase.enabled and storage.snapBase.shadowWrite.
Master's gen-config-docs.mjs now requires desc_ja on every seed entry, so the
docs-sync CI job fails at the 'generate' step (exit 1) before the drift check.
Restored both lines verbatim from the pre-rebase PR head; generate now exits 0
with no drift and docs-coverage stays green.
…— pre-merge gate:local + full-CI validation

# Conflicts:
#	src/server/prepare-responses.ts
@ranxianglei ranxianglei changed the title integrate: #5901 (snapshot plane) + #6564 (msgid delivery) — pre-merge validation [PR] integrate: #5901 (snapshot plane) + #6564 (msgid delivery) — pre-merge validation Nov 14, 2026
@github-actions

Copy link
Copy Markdown
Contributor

📦 Built Package Artifact

Branch: integrate-6564-5901 (3a68dcb)

Latest build — install this exact version

npm install -g billion-context@0.0.9939-pr.6680.1f2cfd0

This exact version is immutable and reproducible. The rolling pr tag tracks the newest test build across all PRs.

Recent builds (commit → version)

commit version build
3a68dcb billion-context@0.0.9939-pr.6680.1f2cfd0 build

Download artifact

  1. Download the artifact from the Actions run
  2. Extract the tarball and install:
tar xzf billion-context-pr6680.tgz
npm install -g package

Install instructions are refreshed on each push.

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

[bot] 🏷

查重 (dedup): Not a duplicate. This is the unique pre-merge integration vehicle combining the two open PRs #5901 (snapshot plane) + #6564 (msgid delivery). No other PR/issue does this combined validation.

标签: dsh pi opencode omp (msgid delivery spans all four clients) · scope/wiring (the cross-client msgid wire delivery is the dominant behavior change; #5901's snapshot tables are default-off storage) · enhancement · p2 (combined with #6564's GA pin rollout wire behavior).

What I did — independent validation of the combined tree at head 3a68dcbbad (this is a DRAFT validation vehicle, not a merge target):

  1. Merge resolution correct. The only real conflict was src/server/prepare-responses.ts. Verified by reading the code: [PR] msgid Delivery: Proposal #6539 Full Implementation (Publish·Record·Use, dsh/pi/omp/opencode × Four Protocols) Consolidat | msgid 递送:提案 #6539 完整落地(发·记·用,dsh/pi/omp/opencode × 四协议)合成一个 PR #6564's inline responses-wire pin is subsumed by [PR] storage: snapshot table compression foundation P0+P1 (schema v12 4-table + shadow write, all off by default, zero behavi | storage: 快照表压缩底座 P0+P1(schema v12 四表 + 影子写,默认全关、零行为变化) #5901's runStage0 — a single incomingMsgIds read (line 116), one runStage0 call routing it (313–321), the pin applied exactly once (322–325), and renderMsgs = s0.kernelInput ?? maybeSnapshotView(...) (355). No double-pin, no duplicate import, no duplicate const incomingMsgIds. Because the resolution folds [PR] msgid Delivery: Proposal #6539 Full Implementation (Publish·Record·Use, dsh/pi/omp/opencode × Four Protocols) Consolidat | msgid 递送:提案 #6539 完整落地(发·记·用,dsh/pi/omp/opencode × 四协议)合成一个 PR #6564's pin into [PR] storage: snapshot table compression foundation P0+P1 (schema v12 4-table + shadow write, all off by default, zero behavi | storage: 快照表压缩底座 P0+P1(schema v12 四表 + 影子写,默认全关、零行为变化) #5901's runStage0 (the surviving mechanism), the result is merge-order-independent.
  2. Typecheck clean — tsc --noEmit exit 0 (kernel + pi-subagents built first).
  3. Targeted suites green (0 fail):
  4. Remote CI: 0 failed so far; the gate lane = success (plus architecture-docs / version-guard / kernel-guard / lifecycle-ledger-guard). 9 guard/docs lanes still running (test-import-graph, plan, docs-coverage, lane-guard, pi-subagents-guard, config-surface-guard, docs-sync, lock-sync, unused-exports).

gate:local / real-agent note: the body's "gate:local in progress (Lanes A/B/U)" is the operator's local last-mile real-model gate (needs real dsh + SGLang + relay infra) — I did not run it here; it's an operator-infra gate, not a code signal, and the CI gate lane already passed remotely. Likewise the body's "real-agent harnesses (dsh + pi)" claim needs real dsh + model infra, which I did not independently re-run.

8问 (结构化结论):

  1. 是什么问题: 不是修 bug,是预合并验证载体——把 [PR] storage: snapshot table compression foundation P0+P1 (schema v12 4-table + shadow write, all off by default, zero behavi | storage: 快照表压缩底座 P0+P1(schema v12 四表 + 影子写,默认全关、零行为变化) #5901(快照平面)+ [PR] msgid Delivery: Proposal #6539 Full Implementation (Publish·Record·Use, dsh/pi/omp/opencode × Four Protocols) Consolidat | msgid 递送:提案 #6539 完整落地(发·记·用,dsh/pi/omp/opencode × 四协议)合成一个 PR #6564(msgid 递送)两个在途 PR 合到 master,验证两者能一起干净落地、全绿。
  2. 什么原因导致的: 不涉及(无缺陷;这是验证,不是修复)。
  3. 影响范围与严重程度: 载体本身不直接面向用户;合并树中 [PR] storage: snapshot table compression foundation P0+P1 (schema v12 4-table + shadow write, all off by default, zero behavi | storage: 快照表压缩底座 P0+P1(schema v12 四表 + 影子写,默认全关、零行为变化) #5901 全部 flag 默认关(零行为变化),[PR] msgid Delivery: Proposal #6539 Full Implementation (Publish·Record·Use, dsh/pi/omp/opencode × Four Protocols) Consolidat | msgid 递送:提案 #6539 完整落地(发·记·用,dsh/pi/omp/opencode × 四协议)合成一个 PR #6564 是 msgid GA pin rollout(线上 wire 行为)。风险取决于两个源 PR,载体只提前暴露合并冲突。
  4. 怎么修复的: 不涉及(无修复;验证的是合并正确性 + 全绿)。
  5. 是否有回归问题: 独立验证合并解析正确(prepare-responses.ts 单 pin、无重复 import/const、renderMsgs 走 s0.kernelInput ?? maybeSnapshotView),typecheck 干净,快照/stage0 95/95、msgid/envelope/fork 146/146 全绿,CI gate 已过、0 失败。未发现回归。
  6. 是否破坏缓存: 不涉及(载体不改缓存策略;[PR] storage: snapshot table compression foundation P0+P1 (schema v12 4-table + shadow write, all off by default, zero behavi | storage: 快照表压缩底座 P0+P1(schema v12 四表 + 影子写,默认全关、零行为变化) #5901 快照平面默认关)。
  7. 是否改了设计方向: 不涉及(合并两个既有 PR,不引入新设计)。
  8. 是否引入了新的配置: 载体本身不新增配置;[PR] storage: snapshot table compression foundation P0+P1 (schema v12 4-table + shadow write, all off by default, zero behavi | storage: 快照表压缩底座 P0+P1(schema v12 四表 + 影子写,默认全关、零行为变化) #5901 的 storage.snapBase.* 快照 flag(默认关)与 [PR] msgid Delivery: Proposal #6539 Full Implementation (Publish·Record·Use, dsh/pi/omp/opencode × Four Protocols) Consolidat | msgid 递送:提案 #6539 完整落地(发·记·用,dsh/pi/omp/opencode × 四协议)合成一个 PR #6564 的 msgid GA pin rollout flag 都是源 PR 自带,非本载体引入。

判定: 合并树在我能本地验证的范围内全绿(编译 + 241/241 定向测试 + CI gate 已过、0 失败),合并解析正确且与顺序无关。等剩余 9 个 CI guard/docs 车道转绿后,owner 可按任意顺序合并 #5901 和 #6564。我会确认最终 CI 状态。

@ranxianglei

Copy link
Copy Markdown
Owner Author

🤖 Powered by ework · qwen3.8-27b

🔶 需要决策

背景: 合并验证树(#5901 快照平面 + #6564 msgid 递送,head 3a68dcbbad)CI 红——#6564 把 pluginLaneServerHistory 灰度从 0% 翻到 100%(GA,owner 2026-11-13 拍板)后,tests/opencode-rolling-cache-marks.test.ts 3 个测试失败。本地验证矩阵钉死根因:非 #5901+#6564 集成冲突,是 #6564 自身回归(#5901 不碰该开关)。

待决问题: #6564 的 pluginLaneServerHistory GA 与现有 rolling-cache-marks 测试冲突,怎么处置?

选项(各带代价):

推荐: A + 测试微调。252/270(标记丢失)是真回归(缓存标记应保留,与身份方案无关);287(ref 漂移)像测试需按 KDD #13 更新。

不决定的默认行为: #6564 CI 红不可合并(owner-only merge);验证载体使命(证明两 PR 一起绿)未达成,需先处置 #6564 回归再重跑。


CI verdict (full detail):

The 11 test shards finished: 3 failed (ubuntu 22/2, ubuntu 24/2, windows 22/1), all on the same 3 tests in tests/opencode-rolling-cache-marks.test.ts:

  1. continued content IDs can be replaced by client pins without losing the new mark — expected 1 breakpoint, got 0 (rolling cache_control mark LOST)
  2. rolling marks follow client-pinned identities on seed and delta — expected 1 breakpoint, got 0 (mark LOST)
  3. rolling cache_control: document payloads stay exact when marks move — ACP tag refs shifted +1 (incremental lane m00002/m00003 vs full-resend baseline m00001/m00002)

Root cause (verified locally): #6564 flips pluginLaneServerHistory from 0%→100% (GA) in src/rollout.ts. Verification matrix (tests/opencode-rolling-cache-marks.test.ts per sha):

sha description result
9495c76 master 12/12
eb0670d #5901 alone 12/12
52e54c7 #6564 alone 9/12
3a68dcb combined 9/12
52e54c7 + flag=0 #6564 + BILI_ROLLOUT_PLUGIN_LANE_SERVER_HISTORY=0 12/12
3a68dcb + flag=0 combined + flag=0 12/12

NOT an integration conflict — it's a #6564 regression from the flag GA. #5901 neither causes nor fixes it.

6 extra Windows failures = flakiness (not a regression): windows 22/1 also showed 6 failures in tests/dsh-native.test.ts (4) + tests/issue2832-preflight-deterministic-empty.test.ts (2), all failureType: cancelledByParent / Promise resolution is still pending but the event loop has already resolved (Windows event-loop timing cascade). Both files pass locally on #6564 (76/0, 4/0).

8问 (结构化结论):

  1. 是什么问题: 合并验证树 CI 红——3 个 rolling-cache-marks 测试失败(标记丢失 + ref 漂移)。
  2. 什么原因导致的: [PR] msgid Delivery: Proposal #6539 Full Implementation (Publish·Record·Use, dsh/pi/omp/opencode × Four Protocols) Consolidat | msgid 递送:提案 #6539 完整落地(发·记·用,dsh/pi/omp/opencode × 四协议)合成一个 PR #6564 把 pluginLaneServerHistory 从 0% 翻到 100%(GA),开启服务端正史渲染;该渲染在 client pin 场景下丢 rolling cache_control 标记,且 ACP tag ref 漂移。本地矩阵钉死:关开关即 12/12 绿,[PR] storage: snapshot table compression foundation P0+P1 (schema v12 4-table + shadow write, all off by default, zero behavi | storage: 快照表压缩底座 P0+P1(schema v12 四表 + 影子写,默认全关、零行为变化) #5901 不碰该开关。
  3. 影响范围与严重程度: 仅影响 GA 后默认开的 opencode 插件车道;client pin + rolling cache mark 场景下缓存标记丢失(前缀缓存失效)。p2(能用但特定条件下有问题)。
  4. 怎么修复的: 本任务只验证不修(修 [PR] msgid Delivery: Proposal #6539 Full Implementation (Publish·Record·Use, dsh/pi/omp/opencode × Four Protocols) Consolidat | msgid 递送:提案 #6539 完整落地(发·记·用,dsh/pi/omp/opencode × 四协议)合成一个 PR #6564 是独立 PR)。推荐 A(修渲染 bug)+ 测试微调;备选 B(更新测试)/ C(回滚 GA)。
  5. 是否有回归问题: 是——[PR] msgid Delivery: Proposal #6539 Full Implementation (Publish·Record·Use, dsh/pi/omp/opencode × Four Protocols) Consolidat | msgid 递送:提案 #6539 完整落地(发·记·用,dsh/pi/omp/opencode × 四协议)合成一个 PR #6564 自身引入(非集成冲突)。关开关即恢复。
  6. 是否破坏缓存: 是——rolling cache_control 标记丢失 = 前缀缓存断点丢失。
  7. 是否改了设计方向: 是——[PR] msgid Delivery: Proposal #6539 Full Implementation (Publish·Record·Use, dsh/pi/omp/opencode × Four Protocols) Consolidat | msgid 递送:提案 #6539 完整落地(发·记·用,dsh/pi/omp/opencode × 四协议)合成一个 PR #6564 把服务端正史渲染从实验(0%)转 GA(100%),消息身份从 content-hash 转向服务端 seq(KDD fix: cache health — remove condense, fix isTitleGen, preserve cache_control #13);与现有测试假设冲突。
  8. 是否引入了新的配置: 无新配置;是已有灰度键 pluginLaneServerHistory(env BILI_ROLLOUT_PLUGIN_LANE_SERVER_HISTORY)默认值 0→100。

判定: 合并树未全绿(3 红,根因 = #6564 flag GA)。验证载体使命未达成。需 owner 决策(修 bug / 更新测试 / 回滚 GA)后重跑。不合并、不关闭(owner-only)。

… the msgid GA (#6539)

The pluginLaneServerHistory GA (this PR, 0% -> 100%) turned on the
canonical-id/seq stamps on the incremental lane's re-entry body for every
plugin session. When that body carries the client's rolling cache_control
marks (opencode's cacheControlRetouch channel), the stamps make the kernel's
first-sight pass re-render the marked content into <acp>mNNNNN</acp> markers,
which drops the wire-level cache_control breakpoints and drifts the ACP refs.

Gate the stamps on the mark channel: coreToAnthropic splices entry.marks into
the assembled items, so a non-empty marks map is the precise signal that this
turn's body carries marks. Mark turns now re-enter exactly as they do with the
rollout off (no stamps) — the pre-GA behavior that always passed — while
non-mark turns keep pinning. The lane still owns the canonical ids.

Tests: opencode-rolling-cache-marks 12/12 (was 9/12); msgid/envelope/fork
148/148; incremental-lane suites green; dsh real-machine envelope e2e ALL PASS.
@ranxianglei

Copy link
Copy Markdown
Owner Author

集成验证载体已被 #6785 取代(同树 + 前向合并 master 解 store.ts 冲突 + gate 全开 lane)。#6785 已含本 PR 全部集成验证结论。本 draft 关闭。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant