Repository navigation
[PR] msgid delivery × snapshot plane: unified merge (absorbing #6564 + #5901 + #6680, including gate fully-open lane local ve | msgid 递送 × 快照平面:统一合并(收编 #6564 + #5901 + #6680,含 gate 全开 lane 本地验证) - #6785
Conversation
…6539) Complements the client-side envelope wiring (#6532 shipped dsh openai+ anthropic). This is the 'record' half of the msgid lifecycle made whole across every wire: - prepare-google / prepare-responses: read + strip the biliMsgIds envelope before it can reach the upstream, then pin behind the same rollout gate as the openai/anthropic twins. A length/shape mismatch is rejected (bail) at the pin — never leaked, never mis-pinned. - handle.ts: a non-body id source (x-bili-msg-ids header / loopback harvest) now stamps the parsed body on EVERY wire, not just openai/anthropic, so the four pin-consumers all see it uniformly. - rollout: pluginLaneServerHistory 0% -> 100% (GA, owner ruling 2026-11-13). No client calls the co-gated /__bili/plugin/render endpoint yet, so flipping only activates pinning. Roll back with BILI_ROLLOUT_PLUGIN_LANE_SERVER_HISTORY=0. Three tests that pinned the old default-off contract now force the off state explicitly (env=0) instead of deleting the var.
Principle 1 of the four-principles proposal (discussion #6262): identity, not bytes, decides which prefix a fork inherits. The plugin-protocol channel (dsh/omp/pi declared forks) gains an optional msgid track: - a host-pinned parent exposes the host's stable message ids AS its snapshot rawIds, so the child may present the host id of each matched prefix message (prefixMsgIds) plus the hash-run length (driftFrom); - the server accepts a byte-drifted prefix only when every presented id equals the parent's rawId at its slot; the receipt is marked status "drifted" (or "expanded+drifted") with the verified driftFrom echoed; - a gate failure never fails the fork: the byte-verified strict fork proceeds exactly as before (no 409); malformed shapes are 400; parents without host-pinned ids make the track a no-op (byte path unchanged). Client: lanes inject their existing parity-pinned collectors (dsh hostMsgIdsOf; pi/omp laneMsgIdsForWire on the anthropic wire with the same content-pinnable gate the incremental pin uses); fork-adopt performs only the core-alignment length lock — no new validator copies, msgid-rule-parity matrix untouched. opencode-v2 is not wired in this PR (byte track, follow-up). Tests: tests/issue6262-fork-msgid-track.test.ts (8) — client prefix extension across a drift cell, the byte-fallback arms, a full-chain proxy round trip, and the server gate (drifted receipt / strict fallback / 400 arms). Docs: msgid-infrastructure.md consumer 6, plugin-protocol.md section 8, architecture.md.
Carrier ① (the envelope body field) had zero client wiring: the server consumes biliMsgIds on both wires (prepare-openai/anthropic, gated on pluginMode + the pluginLaneServerHistory rollout), but no client ever stamped it. dsh's stamping lived only on the anthropic-wire incremental lane (#4491), so every openai-wire full-resend turn — dsh's main path — arrived id-less and the store fell back to content-derived digests, which in turn leaves the fork msgid identity track (#6293) with nothing to match. maybeStampEnvelopeMsgIds now stamps full-resend model bodies (both wires) after the incremental lane had its chance. Resolution never re-implements the host's wire transforms: the pi-ai lockstep walkers (laneMsgIdsForWire / laneMsgIdsForOpenAiWire — the SAME functions the pi/omp lanes pin with) prove wire-element ⇔ host-message pairing by order-preserving fingerprints, bail all-or-nothing; the kernel's own converters then prove core-alignment via the #3642 wireIndex back-pointer (every non-leading wire element must own exactly one core — parallel tool calls / reasoning+text / text+image parts split one host message into N cores, and N cores cannot share one host id). System-role entries are excluded (they ride the hoisted/folded leading system, never 1:1 to a non-leading wire element). Any bail leaves the body byte-identical → content-derived ids, exactly today's behavior; with the rollout off the server strips the field, so default installs are untouched.
#6525 B7) dsh's flat host messages differ from the pi-ai shape the shared walker's host-side fingerprint assumes in two places (role tool vs toolResult, block tool-call vs toolCall), so every tool-bearing turn bailed the all-or-nothing envelope and the turn-1 host-pinned rows churned back to content digests on the next full-resend. Normalize dsh's shape for fingerprinting only, in the dsh resolver: the envelope ids still come from mm.id and the wire-side pairing is untouched, so no mis-pin is possible. Found via a real-dsh e2e dump (after turn 1: live rows 11/11 content digests). Now: live rows 11/11 host-pinned UUIDs. Two regression tests pin the accept (a dsh tool turn core-aligns and stamps) and the retained bail (parallel tool_calls still refuse, one host id cannot cover N cores).
…6525/#6539) The server pins host ids against the POST-HOIST core count (prepare-openai validates ids.length against openaiToCore(parsed).length, which hoists the contiguous leading system/developer run out of the fold space). opencode V2's x-bili-msg-ids header was stamped per WIRE message, INCLUDING the leading system, so its count always exceeded the core count on the openai wire and the pin was rejected (a safe content-derived degrade — but opencode V2 never actually pinned on the openai wire). Core-align the V2 extractor: skip the contiguous leading system/developer run before collecting ids so the header count equals the post-hoist core count. No-op on the anthropic wire (system is a separate field, never in messages); mid-conversation system messages keep their ids; a leading-system-only turn degrades (no empty header). Whole-turn poison contract unchanged.
…#6539 review) Dual-agent review of the #6539 integration found the egress gap: a stamped full-resend body rode biliMsgIds/biliLedgerSeqs out through the verbatim forward branches that bypass the four fully-processed prepares — strict-schema upstreams 400 'Extra inputs are not permitted', and #661 body fingerprinting breaks. - server.ts: protocol-neutral scrubMsgIdFields (byte-identical pass-through when the fields are absent, so #661's contract is untouched in the normal case) + prepareCountTokens egress hygiene. - handle.ts: scrub applied on every verbatim branch (unattributed passthrough, bypass, v1internal background, commandcode unconvertible, no-messages fallback, side-request body, auto-mode forge, final fallback). The #1884 signed-passthrough escape hatch is deliberately NOT scrubbed — the body-covering sdk-hmac-sha256 signature would break; byte-untouched wins. - dsh-native.ts: resolveEnvelopeMsgIds now calls the shared coreAlignEnvelopeIds instead of the cherry-pick'd inline copy (review M1); the fork-adopt msgids use the SAME core-aligned resolver (m1: the old per-wire-message hostMsgIdsOf could never pass the adopter's msgIds.length===core.length lock); explicit incremental-lane guard (m2); ENVELOPE_MSG_IDS_FIELD hoisted to module level with a parity unit test (n2). - opencode-v2-msgids.ts: comment corrected to the actual responsesToCore sysDevHead / anthropic separate-field rules (m3). - non-ASCII note text removed (n1). New: tests/issue6539-verbatim-scrub.test.ts (7). Regression: msgid/envelope/fork unit 70/70 + 28/28 + 22/22; wire e2e google+responses 17/17; plugin-fork 52/57 (5 pre-existing cold-restore fails).
) opencode 1.x's ONLY host-id seam is experimental.chat.messages.transform (harvested per-session by harvestMsgIdsTransform); the wire body only exists at fetch time. So the alignment runs at the shared fetch chokepoint via a new state.beforeSend — proving the harvested turn POSITIONALLY against the ACTUAL wire (alignedMsgIdsForWire), never a verbatim copy of the stored list: a same-session A-transform/B-transform/A-fetch interleave degrades to content-derived instead of mis-pinning. - anthropic-wire only; a turn pins only when every message lowers 1:1 to one core (pure-text user/assistant) — tool/reasoning parts or any count/role/ text drift drop the whole list. The server re-proves count parity + canonical shape and strips the field before any upstream forward (#2821). - Also feeds the shared watermark slot (pfCaptureIds) with the SAME proven list — this replaces the DEAD call that sat in harvestMsgIdsTransform (it fed host objects {info,parts} to a wire-shape proof, so it could only ever clear the slot; V1 watermark was count-only with no tail-ids). - Guards: never restamp an incremental-lane (diverted) turn, never clobber an already-present x-bili-msg-ids (a V2 request stamps its own), budgets match the server parser (4096 ids / 16KiB). New: tests/opencode-v1-envelope.test.ts (9).
…6539) Boots a real startServer + fake upstream (no agent) and drives the exact envelope the dsh/pi/omp/opencode clients stamp: A. normal plugin lane — biliMsgIds pins every host id into the store (the plugin snapshot's rawIds come back as the host ids, not content digests) and the field never reaches the upstream. B. verbatim (bypass) lane — the same stamped body is scrubbed before the forward (the F1 review fix), so biliMsgIds never reaches the upstream. 2/2 pass; typecheck clean.
The dsh/pi/omp full-resend envelope (biliMsgIds) only covered the openai and anthropic wires; a /responses request (body.input) safely degraded to content-derived ids. This adds the responses dialect so all three host agents carry host msgids on every wire the proxy can pin. - pi-msgids.ts: laneMsgIdsForResponsesWire + respWireFingerprint/respUserParts (lockstep pairing via the shared makeEntryTake/partsLowerThrough; conservative bail on mid-history system, opaque items, reasoning, and any item the kernel would split into N cores). coreAlignEnvelopeIds now accepts "responses" and proves 1:1 against the kernel's responsesToCore wireIndex — the perEl check is the final no-mis-pin safety net. - dsh-native.ts / pi.ts: resolvers accept the responses wire (wire = parsed.input). - tests: 11 new cases (1:1 tool round, lead-system hoist, multi-part/mid-system/ opaque/reasoning bails, dup-id bail, resolver pairing with and without lead). Regressions green: pi-omp 34/34, dsh 15/15, opencode-v2 10/10, fork-track 8/8, msgid full suite, wire e2e google 4/4 + responses 13/13.
… write) Implements the snapshot-table compression foundation from discussion snap_rows / snap_log / snap_ckpt, schema v12), the single-transaction commit primitive with S4 checksum verification, and the P1 shadow write that derives the plane from final DB state after each ingest. P0 (zero behavior change, flag default OFF): - schema v12: the four plane tables + idx_l1_ledger, created by migrateSchema (from < 12) and fresh installs; purely additive, rollback = DROP the four tables + user_version rewind (T1 pins the byte-identical legacy dump in both directions). - src/storage/snap.ts: the commit primitive (applySnapCommit / applySnapInTransaction) — events apply in order, one snap_log entry per event with the post-apply S4 aggregate, seq strictly monotonic past max(log max, ckpt_seq), optional fold into snap_ckpt (three- piece meta: kstate verbatim + coverage mapping + S4 aggregate, log truncated at the fold). Read-side rebuild (readSnapView) recomputes the aggregate and kills the plane for the sid on any mismatch (R1, sticky). L1 identity ledger (snapL1Record): issue/arm/echo/hit/wait, hard-refuse on a raw-id → ledger conflict (R3). GC releases rows only after their cover vanishes (invariant iv). - two-level flags: storage.snapBase.enabled (master) ⊃ storage.snapBase.shadowWrite (lane), config-file only (#2030), resolved in knobs (fileSnapBase) and re-asserted at the store. - /__bili/status gains a snaplane field (flag state + counters + dead latches); sessionStorageStats gains the plane dimensions. P1 (shadow write, read side still never serves): - after each committed ingest, one store transaction records the L1 decision and applies the derived events: steady-state delta appends take a fast path (no full scan); everything else is a final-state diff over live rows + block coverage (cast/update/ delete/compact, pos-ordered, deterministic). A round that cannot be addressed produces no events (R2); a crashed gap self-heals on the next save because the derivation is a final-state diff. - fold at 128 log entries or on any compact; the folded plane is checked against the legacy view (A/B oracle) — a mismatch kills the plane for the sid. Covered positions are legitimately inactive in the plane and are accounted for in the ckpt coverage map. - deleteSession cascades the four tables (no orphans, T6). Tests: tests/issue5874-snapbase-p0.test.ts (T1/T1b/T2/T2b/T2c/T2d/ T6 + flag nesting) and tests/issue5874-snaplane-p1.test.ts (T3 B1 trap nail / T4 A/B incl. crash gap / T7 coverage map / T9 bounded growth). Schema-head assertions in the existing migration suites bump 11 → 12. Timing dependency (#5855): the unmerged fastpath re-land also numbers its two table groups v12/v13; whichever lands second renumbers its block mechanically (independent CREATE IF NOT EXISTS shapes). mysql backend: the plane is sqlite-only for now (the async backend opens without the flags). Refs discussion #5874.
…n open sqlite files (#5874)
…5874) snapGcCoveredRows removed rows outside the commit primitive without updating the anchor they aggregate over: after an ordinary host-side rewind (raw-tail historian sweep retires rows, the diff flips them active=0 with cover_ref='deleted'), the immediate release left the last log sumAfter covering rows that no longer exist, so the next readSnapView tripped R1 on a healthy plane and the sticky dead latch kept the session's plane closed until process restart. Removal now re-anchors in the same transaction (a pure-reanchoring fold at the current log tail), preserving the intended release timing. Also adds the missing P1 fast-path coverage: every prior P1 test saved through planIngest(full), so snapFastAppendEvents never ran in CI. The new suite drives the real watermark/delta machinery (UnifiedShadow.record): steady appends ride the fast path ('append' kinds), and a rewound tail voids the delta, converges byte-for-byte through the diff, and keeps the plane alive across the release round (regression pin for the R1 latch).
The derivation reads final DB state, so how the legacy rows arrived (complete log vs delta) is irrelevant to it; the param was threaded but never read.
Temporary soak change (revert before merge per owner instruction): when the config file is silent, the snapshot-table plane now defaults to enabled + shadowWrite (was off). Every hermetic test and the local real-traffic gate therefore runs the shadow write path on all sessions; the config file can still pin it off explicitly. Spots to revert (all marked 'SOAK' in comments): - src/storage/store.ts: field default + open() fallback - src/server/admin.ts: snapplane status default - tests/issue5874-snapbase-p0.test.ts: default-expectation asserts - tests/issue5874-snaplane-p1.test.ts: crash-gap store pins OFF
Per-table storage relationship map for the snapshot-table foundation (#5874, this PR's schema v12): every table card shows what it stores, when it is written, who reads it, and which feature it serves — ingest, shadow-write, fold, egress (legacy today / plane in P2), hydration, deleteSession cascade, GC, and the /__bili/status surface. Registered in docs index.
…ries The design doc pair (proposals/snapshot-compression-foundation) moves from the proposal branch into the implementing PR, per the docs-centralization standard: - bilingual README.md / README.zh-CN.md + 4 arch diagram pairs (svg+png) + the arch-migration generator script - status line updated: P0+P1 implemented in #5901 (schema v12, shadow-write soak, zero egress change); P2 egress cutover still pending owner ruling - §5 version-stacking corrected to reality (landed v12 on v11 master while #5855 is still open; later-landing side renumbers mechanically) - §9 keys finalized (storage.snapBase.enabled / .shadowWrite, config-file only) - §12/§13 notes: P0+P1 shipped as one PR; #5855 prerequisite not waited on - cross-links to the per-table map docs/storage-relations.{en,zh}.svg - registered in docs/index.md + docs/index.zh-CN.md
…AGE_SCHEMA_VERSION (#5874)
…th (#5874) docs-sync red on fb17874: the committed generated rows claimed default 'on (soak; off in release)' while the canonical seeds (website/config-reference/process.yaml) say default false — the generated artifact had drifted from its source. Rerun tools/gen-config-docs.mjs generate and commit its output (2 rows per file, en+zh). Note for reviewers: the #5901 soak (dab9292) makes the code default ON when the config file is silent until the owner-instructed pre-merge revert; this doc describes the seed state, which equals the post-revert code.
…hema versions Field incident (2026-11-08, shared index.db): the unmerged #5855 re-land consumed schema v12+v13 (pf_* planes) into a database this branch's v12 block had never seen. migrateSchema's early return (from >= STORAGE_SCHEMA_VERSION) skipped the DDL, so the first save died with 'no such table: l1_world' and every session's plane was marked dead (R2, by design — but the root cause was the version gate). The DDL is pure CREATE ... IF NOT EXISTS (no backfill, no rewrite), so it is safe at any from-version: ensure the four tables by TABLE-EXISTENCE PROBE before the early return (the discipline v2/v5/v6/v8 already use), instead of gating on user_version < 12. Both merge orderings now coexist; the final master version number stays a mechanical merge-time hygiene step. user_version is never downgraded (the early return leaves it alone). T1c pins the exact field state: v13 database with sibling pf_* tables and no snap tables → open → snap tables created, version stays 13, sibling rows untouched, legacy dump byte-identical, plane functional. 14/14 suite green, 35 storage regressions green, tsc clean.
…CI)
readPkgVersion is polled every 500ms by waitFor in the registry/advisory e2e suites; a poll landing inside the updater's remove-and-reinstall window hits a transiently absent package.json and the raw readFile throws ENOENT out of the predicate, killing the subtest at ~1.6s despite a 60s budget (ubuntu red job 128697545236; windows green on identical code = race). All four copies of the helper now return null on ENOENT ('not yet', not an error); one-shot assert sites still fail loudly on absence.
…-for-byte guard (#5874 P2) The outbound compressed context is now assembled from the verified snapshot plane (mapEgress) instead of the in-memory history, on all four protocol lanes, guarded by a tripwire: the plane view is built first (anchor summaries at each covered region + surviving rows + the client tail, the same seam walk as the inbound view), the lane's exact post-processing is reapplied, and ONLY a normalized exact match switches the outbound bytes. Any mismatch keeps the legacy bytes; after three mismatches the session is sticky-excluded for the process lifetime. Every degrade reason surfaces on /__bili/status under snaplane.egress. - new src/server/snap-egress.ts; wired into prepare-{openai,anthropic, google,responses}.ts right after the legacy processedMessages - store accessors: UnifiedStore/UnifiedShadow/SessionStore.snapEgressView - config: storage.snapBase.egress / egressCohorts (soak default ON at 100%, release cut flips to opt-in) - admin status wire: snaplane.egress {cutover,fallback,planeMiss, excludedSids,degradeReasons} - tests: 16 new cases (view assembly incl. reply-lag/rewind/remint/gap seams, tripwire normalization, anchor rendering parity, gates + sticky exclusion); docs en/zh updated (configuration, proposal record, index)
…stem role (type check over tests)
…ed prefix rows The client harness wraps every resent message in a per-RESEND envelope tag (<acp tokens=.. time=..>mNNNNN</acp>) whose time attribute changes each turn, so the snapshot plane can never store it byte-for-byte. That made the tripwire fire on every folded dsh turn (shape@0:text) even though the plane content was identical, and sticky-excluded those lanes from the cutover. The egress view now pairs each surviving plane prefix row with the identity- matched message from THIS resend (id first, then envelope-stripped signature) and adopts its exact tagged text, but only when the pair is content-equal modulo the envelope — otherwise the plane text stands and the byte-strict tripwire judges it. Envelope changes between turns still seam-match because both bySig indexing and the seam walk compare envelope-stripped signatures.
…, lock sync, internal types - website/config-reference/process.yaml: seed storage.snapBase.egress / storage.snapBase.egressCohorts and regenerate docs/configuration*.md (config-surface-guard, docs-sync) - reference/architecture.md: register src/server/snap-egress.ts (architecture-docs) - package-lock.json: sync the kernel workspace to 0.0.189 (lock-sync) - snap-egress: drop export on the file-local SnapEgressDeps / SnapEgressStatusWire interfaces (unused-exports)
…-refs pass over the plane view The <acp tokens type time>mNNNNN</acp> tags are NOT client resend data — they are system metadata rendered by the kernel's render-refs node (the LAST pipeline node) over turn.messages. The earlier per-resend envelope adoption was therefore the wrong model. The plane view now reaches byte parity by re-running the kernel's own renderer (renderVisibleRefs with the deterministic default counter) over the assembled view, using the same post-turn state the lane already holds: refs from state.messageRefs, immutable first-sight times hydrated from state.tsStamps, strategy from the lane's renderTags knob, timestamp mode from resolveMessageTimestamps (env > file > default ON). Render throws degrade honestly (render-error, legacy bytes kept). Lanes pass renderTags; T18a/b rewritten to assert kernel-render parity and first-sight time stability; T18c retitled to the content-edit divergence case. 19/19 + regression suites green.
|
(ai: sglang-gpu0/qwen3.8-27b) seg 4b 完成:出站会话从 op-log + content CAS 重建( 延续 #7398 的 op-log 出站视图,seg 4a 验证了「从日志+ CAS 重建 == 旧路径字节一致」,这一提交把影子升级为正式服务:
|
(ai: vllm-qwen/qwen3.8-27b) 收到 seg 4b( |
(ai: vllm-qwen/qwen3.8-27b) CI fix — PR #6785 (Windows test cleanup EPERM, pushed as
|
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) CI fix — PR #6785 (flaky Windows test, needs decision)Re-verified live state first. Branch head is Investigation:
Diagnosis: This is a flaky test — a transient 🔶 需要决策 This is a pre-existing flaky test, not a product bug or a regression from the seg 4b commit. Options:
Recommendation: A — a one-line retry wrapper is the smallest fix that makes the test deterministic. Default if undecided: B (no change; wait for the next CI re-run to confirm it's a one-off flake). I have not pushed any change yet — waiting for your call on A/B/C. |
…requests (#7398) The pf handshake lanes (watermark / delta) only need the byte span scan plus the head + tail window fragments, but handle.ts parsed the entire conversation body before evaluating them — 150-600ms of pure waste on a 155MiB plugin-lane resend. A large OpenAI body carrying any pf handshake header now gets a top-level object synthesized from the span scan (all fields except the multi-hundred-MiB messages array, which is never needed whole on a hit); prepareOpenai re-parses the full body exactly once, and only when every lane misses. No pf header / body under 1MiB / unscannable shape / any field-span parse failure = legacy full parse, byte-for-byte the old behavior (fail-soft, never guess). Field-verified: 155MiB full-form lane 14.3-16.9s -> fastpath HIT every turn with stamps; the messages=null marker is exempted from the non-conversation relay gate (a deferral body IS a conversation, its array is just decoded later). Closes the residual-parse item of discussions/7398.
…he sidecars (#7398 seg 4b) The v1 rebuild guard refused any view containing a role=tool row, which made EVERY coding session (every bash/tool result) permanently unservable. The recorded CAS bytes are the full core message JSON, so toolCallId and toolName sidecars are present; the rebuild already runs the same systemToUser -> coreToOpenai -> harden chain, so byte-identity is a structural guarantee, not a per-role hope. The e2e direction asserts were vacuous while they lived: the test sent a custom header name the server never reads (opts.sessionHeader is x-acp-session), all four sessions were anonymous pfa-* ids, and both asserts matched strings that never appear in the log. They now send the session name via x-acp-session and scope: text-B served, tool-B served, both -A controls (rollout off) never served. Fixes #7398
|
(ai: sglang-gpu0/qwen3.8-27b) seg 4b 收尾:tool 行现在也能从 op-log 供数(CAS 本体携带 sidecars),commit 改动
验证:4 个 egress 测试文件 25/25;typecheck 干净;build 通过。 剩余(下一批):真实折叠轮的 survivor id 稳定性实测(folding proxy 开 egressOpLog 跑一条,检查折叠轮 ops 是否为稀疏的 D(区间)+I(摘要) 而非 D+I 全量)。 |
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) CI 确认 —
|
|
(ai: sglang-gpu0/qwen3.8-27b) 性能验证方案:长会话稳态每轮耗时(10w 条消息 / ~155MB 上行 / ~30w token 出口)测什么长会话跑到后期的自然稳态,与模式无关(plugin/proxy 都是这个形状),master 上今天跑够长也是这个形状:
怎么造(增长重放,不用静态语料)之前 bench 没覆盖到这个形状的原因:静态语料要么是纯 10w 条无块、要么是折完的小视图;而真实稳态是长出来的——10w 条原始行 + 几十个分层块共存于同一条上行里。所以按真实轨迹重放:
对比设置(同一条轨迹跑两遍)
同一随机种子、同一消息生成函数、同一折叠触发点,两侧轨迹逐条一致;同一台机器、同一 fake 上游。 指标与验收
步骤快进建态(到 10w 条)→ 稳态测量(10–20 轮)→ master / PR 交叉跑 → 结论回贴到本 PR。 |
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) 收到性能验证方案。CI 侧无阻塞: |
…audit 1. deleteSession/archiveSession listed every v12 table except egress_state/egress_flow — recorded sessions orphaned the op-log rows and their CAS bodies forever. New store.deleteEgressOps() sweeps both tables and returns the CAS refs for release; both call sites wired. 2. buildEgressViewRows putContent-ed every row every turn — putContent refcount-bumps on hit, so each row's refcount inflated +1/turn with the only release at the session sweep. Unchanged rows now reuse the prev casRef (missing ones backfilled, changed rows re-put). Regression test: tests/egress-oplog-lifecycle.test.ts (both fixes, both directions per #3617). Part of #6785.
- NEW docs/pr-6785-unified-storage-and-flows.{md,zh-CN.md}: the full
review-grade report — table structure (schema v12, all 8 new tables +
payload contracts), per-module implementation logic, per-turn data flow
(steady / fold / degrade / lifecycle), the complete switch ledger
(rollout keys, config objects, headers), and the 2026-11-21 ideal-model
incremental bench with honest conclusions.
- REMOVE docs/proposals/snapshot-compression-foundation/ (described the
RETIRED v1 four-table plane; contradicts the shipped v12 shape) and
docs/storage-relations.{en,zh}.svg (5 tables missing, stale cards) —
both were PR-introduced, never on master; the report supersedes them.
docs/index.{md,zh-CN.md} rows swapped; schema.ts comment re-pointed.
- website/config-reference/rollout.yaml: pluginLaneServerHistory default
corrected to 100 (GA), egressServeLog description corrected to the seg
4b reality (self-recording, true swap, whole-turn degrade), new
pluginMsgIdIncremental entry (was undocumented — the dynamic env name
evaded the docs-coverage CI); docs regenerated.
- Stale comment corrections: rollout.ts egressServeLog (no longer 'seg 4a
LOG ONLY' / 'tool rows in v1'), schema.ts v12 note.
Part of #6785.
|
(ai: sglang-gpu0/qwen3.8-27b) The full review-grade walkthrough of this PR is now in the branch as a doc (per the owner's request): docs/pr-6785-unified-storage-and-flows.md (+ zh-CN twin). Coverage: 1) table structure — schema v12 probe-first migration, all 8 new tables with DDL + row-payload contracts, existing-table key columns, MySQL parity gaps; 2) implementation logic per module (slim / snap stage-0 / egress-state engine / egress-record / egress-serve / inbound watermark delta ingest / parse-defer / fastpath watermark+delta arms incl. bugs A/B/C / incremental msgid pin / admin routes); 3) data flow — steady turn, fold turn, the full degrade table, delete/archive lifecycle cascade; 4) the complete switch ledger (rollout keys with defaults+rollbacks, config objects, headers) — incl. the two GA flips (pluginLaneServerHistory 0→100, new pluginMsgIdIncremental default 100). Writing it surfaced two p1 storage-only-grows leaks in this PR's new code, fixed on-branch (commit
Stale docs removed (both PR-introduced, never on master): Also in the report §5: the 2026-11-21 ideal-model incremental bench with honest conclusions — view-shaped wire (~750KB), per-turn cost grows with the cumulative ledger (1w→0.5s … 9.4w→1.8s, the #5744 slow-ingest domain), <1s met at ≤3w; the egress-keys-ON minimal validation (155MB raw wire → 30w-token outbound) is listed as the pending follow-up. |
…ady on PR branch at new location)
[bot] 🏷 (ai: vllm-qwen/qwen3.8-27b) Merge conflict 已解决 —
|
…dows CI flakes) Two mechanical test-hygiene fixes for the 3 red windows-latest shards (run 44864811218, sha 58abab9): 1. tests/egress-oplog-lifecycle.test.ts: the deleteSession test opened a UnifiedStore and never closed it before rmSync(dir) in finally — Windows cannot unlink an open SQLite db (EPERM on rmSync, EBUSY on unlink index.db). Close the store before rmSync, matching the precedent in tests/archive-reference-counts.test.ts. 2. tests/resume-miss-timestamp-seed.test.ts: the earlier e2e test ("resume-inheritance MISS seeds...") left "old:8" (its miss-seed content) in the shared prefixAffinity singleton — its finally forgot only "new", not "old" (re-imported by phase-2 hydrate-on-listening). The next e2e test reuses id "old"; on Windows CI its phase-1 note did not overwrite the leftover, so the flush persisted the leftover (897B = old:8, not 1032B = old:10) and the gate replay classified as prefix-break instead of equal-depth. Forget "old" in both finally blocks so each e2e test starts from a clean tracker.
…ror path The background quick_check worker only closed its read-only SQLite handle on the success path. A throwing quick_check (the corruption case tests/issue4816-integrity-throw-quarantine.test.ts exercises) left the handle open until the worker thread tore down, and a concurrent rmSync of the test's temp dir raced that teardown — EPERM on windows-latest. The error format (EPERM, Permission denied + direct rmSync, no rimraf frames) does not match the CI's cleanup-lock retry shape, so it failed hard. Close the handle on the error path too (best-effort; the verdict is already posted). The success path is unchanged.
Fixed the 2 red windows-latest shards on PR #6785 (final head Shard 24,3 (issue4816 Shard 24,2 (session-gc 八问(中文结构化结论):
收工判定:failure 1 已在代码层修复并本地验证(8/8 + typecheck + build 绿);failure 2 判定为瞬时抖动,依赖本次重跑收敛。最终 head |
The egress flow tables (egress_state / egress_flow) now carry the message BYTES directly instead of a content-CAS hash reference. Content identity is byte equality — no hash, no fingerprint, no pointer to the content table. - schema: add a bytes BLOB column to both tables (probe-first ALTER for dev dbs; fresh dbs get it from the DDL). The tables are #6785-introduced and were never released, so the row model is free to reshape (no migration). - egress-state: EgressViewRow carries bytes (was contentHash+casRef); I/U ops store the bytes in the BLOB column, B rows embed entries (with bytes) in the payload. Missing bytes is a HARD ERROR (pre-#7439 row or corruption) — the caller degrades the whole session to the legacy path. - egress-record: buildEgressViewRows drops the CAS put (no egressPutContent seam); each row's bytes are the message's final core JSON (system row = the outbound system text). The system msgid stays content-addressed (the hash is only the ID, not the content carrier). - egress-serve: rebuildOutboundFromLog reads the bytes straight from the view (no EgressContentStore / CAS lookup); an empty/missing byte payload nulls the whole rebuild (degrade, never guess). - store: drop egressPutContent/egressGetContent (dead); deleteEgressOps just deletes the rows (nothing to release — the bytes ride the rows). - prepare-openai: the serve + shadow paths call the 1-arg rebuild. - tests: rewritten for the self-contained model (contentHash/casRef/CAS gone); new hard-error case for a NULL-bytes row; e2e comments updated. Rollout egressOpLog / egressServeLog stay default 0 (byte-identical when off). Step 1 of the #7439 construction (流水表自包含), on PR #6785; steps 2-8 remain.
…ng 3+5) Step 2 of the #7439 construction, on PR #6785. Ruling 3 (delete the in-memory view cache): - Removed session.egressView (the per-process cache, the 'restart full-rerecord' drift source). The prev view is now read from the flow table (recordEgressTurn's prev param is optional; omitted = read from the flow table, the single source of truth). - prepare-openai uses a LOCAL egressViewThisTurn (only the current turn's recorded view, for the shadow-compare); the next turn's prev comes from the flow table, not memory. Ruling 5 (the degradation ladder — NEVER back to client bytes): - New rebuildOutboundWithLadder: current point first, then walk BACKWARD through the flow table to the nearest healthy point (capped at 100 points). A zero-row view is not healthy (the walk went past the first flow row — log_seq values start at 1). - Whole flow unreadable → session-level recovery: log + admin flag (session.metadata.egressFlowUnreadable) + the request FAILS rather than serving client bytes. - New egressFlowMaxSeq (flow table) for the ladder's starting point. Rollout egressServeLog stays default 0 (byte-identical when off). Tests: 3 new ladder tests (healthy current point / corrupt current point walks back / fully unreadable → null). 31 unit + 35 e2e green.
…-decouple) Conflicts resolved: 1. src/server/prepare-openai.ts: union of fastpath imports (PR's DeltaReanchorError/PF_TAIL_IDS_HEADER/evaluateDeltaResend/parseTailIdsForPin + master's stampOnlyWatermarkTurn). 2. tests/fastpath.test.ts: union of fastpath imports (PR's evaluateDeltaResend/PF_DELTA_HEADER + master's stampOnlyWatermarkTurn). 3. src/agent/pf-channel.ts: kept PR's TAIL_MAX_BYTES byte-budget (supersedes master's MAX_TAIL_IDS count-cap; the byte check is stricter and tailPayloadFor is the single source of truth for both header emit AND full-form suppression). 4. src/server/fastpath.ts: kept BOTH evaluateDeltaResend (PR #6785 delta arm) AND stampOnlyWatermarkTurn (master #7367 stamp-only turn). 5. src/storage/shadow.ts: kept master's fileJsonWriter parameter (new in master, not in PR branch). 6. tests/pf-channel.test.ts: updated #7367 tail-cap test to match PR branch's byte-budget behavior (full slice under 8 KB, not capped to 160).
…ge bytes itself (ruling 7439-1)
The egress op-log no longer references the global content CAS: each I/U
row stores the message's FINAL core JSON in its own `bytes` column and the
system row uses the constant msgid "egress-system". Content identity is
judged by direct byte equality (no hash, no fingerprint fields anywhere in
the egress path), so the two tables replay and serve with nothing but
themselves. The tables were never shipped (v12, probe-first) ⇒ row/payload
format changes freely; local dev DBs upgrade with one probe-first ALTER.
- egress-state: row model {msgid, ref, tokens, type, role?, bytes}; diff
spine unchanged, +U now detected by byte inequality; B base entries
carry bytes inline
- egress-record: buildEgressViewRows drops the CAS put seam entirely
- egress-serve: rebuildCoreFromLog split out lane-neutral (feeds each lane
its own conversion chain — the 4-lane rollout per ruling 7439-4);
rebuildOutboundFromLog stays the openai wrapper
- store: egressPutContent/egressGetContent seams deleted (dead after this
change)
- tests: the five egress files converted to the byte model + new
self-containment regression (3 turns of append/fold/edit leave the
content table EMPTY; delete cascade is plain-row deletion)
Verification: typecheck clean; 126/126 = unchanged vs the pre-change
Step-0 baseline battery (egress family + issue5874 family + fastpath/pf).
…te (ruling 7439-3/5) Ruling 7439-5 (owner strategy decision, discussions/7439 §1): the server egress flow is the SOLE authority over conversation content. This commit wires the last two pieces: - The prepare-layer authority seam (pinned plugin lane): existing rows are swapped to the bytes the server recorded last turn; the client resend is demoted to identity/order channel (per-turn probe) + byte source for NEW messages only. A client byte edit under a stable msgid is a divergence event (logged + warned) — server bytes win and the client converges on its next echo, because the outbound is served from the same log. This overrides the pre-PR 'plugin lane: client-authoritative by design'. Scope is deliberately the pinned plugin lane: proxy clients are tamper-evident by construction (ids are content hashes) and proxy-lane kernel input must stay the raw history (the kernel folds internally). - Degrade ladder (never guess, never client-as-authority): recoverEgressBase hydrates the flow's current point; a structurally corrupt point walks back (bounded) to the newest healthy log_seq and TRUNCATES past it — rows beyond a corrupt point are unusable by definition (no replay can include them), so recovery = flow back to the last healthy point. A failed probe or missing healthy point serves legacy bytes this turn, does NOT advance the flow, and self-heals next turn. The in-memory session.egressView is gone (Step 2) — every turn is a pure function of the tables. tests: egress-authority-ladder (ladder both directions + probe rows), fixtures gain the ladder seam methods; battery 130/130.
The egress plane is no longer OpenAI-wire-only. The non-OpenAI lanes (Responses / Anthropic / Gemini) now go through the same lane-neutral core round-trip via src/server/egress-lane-io.ts: - serveLaneCore — rebuild the session's recorded core view from the flow tables (recoverEgressBase: current healthy point or bounded walk-back + truncate) and hand it to the lane's OWN conversion chain (coreToGoogle / coreToAnthropic / patchResponsesInput). The system row stays out of the served core; each lane builds its own system block live (prefix-cache anchor), exactly like the OpenAI lane. - recordLaneTurn — append this turn's outbound core view as sparse ops; a corrupt log point skips the turn and self-heals next. Per-lane gates: Gemini/OpenAI skip title-gen side requests, Responses skips compaction-generation turns (trigger/prompt form), Anthropic has no transient side requests on this lane (noted inline). The authority SEAM (client resend demoted to identity channel) stays OpenAI-lane-only for now — it rides the stage-0 pinned-plugin infrastructure; other pinned lanes are a later extension (noted in the follow-up list). Rollout keys unchanged (egressOpLog / egressServeLog, default 0 = byte-identical to before); all-lane behavior tested in egress-authority-ladder (helper round-trip + ladder + total-corruption degrade both directions); lane suites (anthropic/responses/google) green with the rollout off.
…layed store The mine-wins replay of the 7439 egress commits replaced store.ts with the old-base version, which predates the #7156 envelopeImport guard (2026-11-21 incident: a pruned envelope render from another writer must not regress the stored block lineage / lifetime stats). Re-applied verbatim from 3ad00e8.
…ev plane (7439 ruling) Every v12 table that never entered master is deleted except the ones the ruling keeps (l1_world, orphan_blocks, state_fields, egress_state, egress_flow). The #5874 v2 snapshot-base plane (fold_journal + egress_snapshot + egress_scope_rev) shipped no behavior yet — its D-lane byte cache was bypassed on the plugin lane and its sole write consumer was this PR — so it is removed wholesale: - schema.ts: three DDL consts + all probes (incl. the covered_ids / ids_json DROP-column and scope_rev ADD-column migrations); v12 keeps l1_world / orphan_blocks / state_fields / egress_state+flow (probe-first, unchanged). - snap.ts: fold-journal + egress-snapshot functions deleted; the L1 world / R3 latch functions (stage-0 identity + pf pin) are kept. - store.ts / shadow.ts / persist.ts: egressCut / egressReadView / appendFoldJournal / appendEgressSnapshot / bumpEgressScopeRev and all seven per-write rev-bump sites gone; delete/archive cascades trimmed. - snapshot-render.ts: the splice (warm serve) and cut blocks gone — the D-lane reverts to live derive, pre-#5874 behavior. - config surface: storage.snapBase (enabled/cohorts/cutEveryKUserTurns) removed from config.ts, knobs.ts, process.yaml + regenerated docs — the plane never shipped, so there is no compat story to keep. - tests: the three plane-only suites retired; 111/111 battery + 108/108 lane suites green, typecheck clean. The byte-level prefix-cache gain the plane would have given is superseded by the self-contained egress op-log, which serves the FULL rendered view from the flow tables on every lane.
…erveLog into egressPlane (#7439 ruling) The egress plane gets a SINGLE rollout key: egressPlane (default 0). ON = the complete new path in every lane — sparse-op recording, serve-from-log (byte-identical swap), and server authority for pinned plugin lanes (rulings 7439-3/5). OFF = legacy chain, zero rows, byte-identical to pre-feature. Both predecessor keys never shipped (default 0, born in rollback state) — no migration. Owner ruling 2026-11-21: '删掉以前的开关 只留下一个开关'. Verification: typecheck 0; 13-file storage/egress/fastpath battery 112/112; e2e codex 4-phase pass (rollout off = legacy path).
|
(ai: sglang-gpu0/qwen3.8-27b) egress 平面收官:五笔重放 + 单开关合并,已推送到分支( 按 owner 裁定(讨论区 #7439 + 2026-11-21「只留一个开关」),本批把 egress 平面收口: 声明:语义重复的实现以这批为准。 远端此前有两笔同裁定实现(
新增两笔(远端没有的):
config surface 声明(AGENTS 要求):退役 验证:本地 typecheck 0;存储/egress/fastpath 13 文件电池 112/112;e2e codex 四阶段全过(关开关走 legacy)。全量回归以远端 CI 为准。 未做(另开跟进 issue):裁定2 的 blocks Shape A(blocks 表仍引用 content CAS 存摘要);content 表「过期」标记。 |
msgid 递送 × 快照平面:统一合并(收编 #6564 + #5901 + #6680)
按 #5874 六段固定链把两条提案线合并成一个 PR:msgid 递送底座原样保留,快照存储模型从旧三表平面整体重写为纯增量 v2——旧方案(#5901 的 snap_rows/snap_log/snap_ckpt 四表)从未真正实现 #5874 的目标(单一存储真源、压缩不绑 session),这次一次做对,不留返工面。
内容
1. msgid 递送底座(原 #6564,原样保留)
四车道宿主 msgid 信封、identity-harvest 自包含采集器(env/灰度双门)、GA rollout、e2e-msgid-envelope / issue6525 / issue6262-fork / openai-msgid-pin / msgid-rule-parity 全套。非破坏:all-or-nothing pin、证明不了降级回内容摘要、只有带信封代码的客户端触发。
2. stage-0 身份层 + R3 门 + M2 增量入站(原 #5901 中保活的部分)
l1_world(身份探针/回声/见证)、R3 身份熔断(10min/3 strikes)、M2 msgid 信封入站、2x2-p3 测试族。
3. 存储模型 = 纯增量 v2(#5874 六段链,替换旧三表)
append-only 底座(复用既有):content CAS、messages(历史改写 = 新 seq 行 + 旧行标记退休,绝不 in-place)、checkpoints(回滚/fork 共用同一水位原语)。
fold_journal(新表,append-only):每个 fold 记一条决策点(边界 seq、covered ids、输入哈希),与 fold 同事务落。unpin 判定是派生不是存标志:fold 的 covered id 仍有存活行 = unpinned(Four storage-layer defects: identity hash contaminated with positional metadata (truncation triggers identity swap) / sw | 存储层四缺陷:身份哈希掺位置元数据(截断即换血)/清扫删除回退历史/渲染接缝图片只比长度/fork 谱系死列 #4390 原地复活时正史仍渲染原文,旧字节仍有效)。
egress_snapshot(新表,缓存非日志):D 车道渲染把服务端正史渲染的稳定前缀物化成一条 CAS 引用行,切点 = fold 水位与最新存活用户消息(天然回滚点)取新者;暖轮渲染逐字节复服、跳过缓存前缀的内容加载。三道相互独立的拒绝门保证缓存诚实:① id 绑定输入哈希重算(同 id 内容搬家即降级)② 跨前缀已 pin fold 门(吃掉前缀的 fold 即降级)③ 拼接行存活检查(账本已不载的缓存行即降级)。任何失配该轮整体回退实时推导,下一轮渲染自修复切点——永不粘死、永不拼半截。
性能瘦身(sessions 行不再背无界大字段,sqlite/mysql 双端同构):
c4102813c):ccr_json列只留 key→hash 信封(KB 级),原文搬进ccr_entries.orig_hash走既有 content CAS(跨会话同字节去重);读侧优先全量旧列(存量行逐字节),瘦信封 + 任一 entry 缺原文 ⇒ 整路回退文件车道,永不拼半截。a03c666b1):reaped block 视图移入orphan_blocks表(一个块一行),列迁移后退役;读侧 = 表行 ∪ 存量列键的并集视图。a03c666b1):九个无界 kernel 字段(tokenSnapshot/tsStamps/lastPassIds/hiddenOrphanRefs/sweptOrphanCallIds/renderDeferredRefs/viewLatches/absorbed/rules)移入state_fields表(一字段一行),kstate_json列只留有限字段;存量会话零行时整列原样生效(隐式一次性迁移,无回填)。4. config 面收窄(§ config surface)
见下方专门章节。
config surface(规范要求的显式报告)
storage.snapBase由四键(enabled/shadowWrite/egress/egressCohorts,随旧平面已不存在)收窄为两键:storage.snapBase.enabled(boolean,默认 false)与storage.snapBase.cohorts(string[],缺省/空 = 全部协议)。BILI_ROLLOUT_*灰度面是 env 层,平面开关按 彻底整治环境变了滥用,除非必要都走配置文件 #2030 纪律只做 config-file 层(无 env)。验证矩阵(head
c28489e6c)a03c666b1上 CI 为准所有新门(gate)按 #3617 纪律带 ± 双向断言:L5 钉死「默认关 = 不切不读」与「cohort 排除 = 同渲染不切」;L2/L3 钉死降级路径字节级等价且自修复。
默认合并影响面与开启方式(2026-11-16 补)
默认为开(合并即生效,全部是存储实现改法,对客户端逐字节透明,± 测试钉死):
c4102813c)ccr_json列只留 key→hash 信封;稳态保存零写a03c666b1)orphan_blocks表(一块一行),列退役(首次保存隐式迁移,零回填)a03c666b1)state_fields表(一字段一行),列只留 14 个有界字段;稳态逐行比较零写e43e430c6)三项均 probe-first 新表(不动 user_version)、sqlite/mysql 双端同构;回滚 = DROP 表 + 代码回退,数据零迁移(读路径永远兼容新旧格式)。
默认为关(出站稳定前缀缓存,D 车道物化前缀):
cohorts= 上游协议范围,取值为四条出站车道:openai/responses/anthropic/google;缺省/空 = 全部协议/__bili/status平面状态;日志降级标记(缓存失配该轮整体回退实时推导,下一轮自愈,永不粘死);egress_snapshot表行(缓存切点)storage.snapBase键;已缓存行惰性失效,无清理操作本次明确不做(列为 follow-up,不塞进本 PR)
(kstate 瘦身 / orphans 拆分 / ccr 镜像收窄 已在本 PR,见「性能瘦身」;mysql 后端同构支持已随各步一并落地——DDL、读写、级联、归档恢复双端一致,seam 测试覆盖。)
分支构成与开关安全
origin/master(26d9bcc64)+ step 0 合并(#6780/#6793 persist 并集)+ step 1(退三表、留身份层3fe1352fb)+ step 2(fold_journale43e430c6)+ step 3(egress_snapshotbd9c467b1)+ step 4(用户消息边界切点22098cc36)+ step 5(config 收窄7e4910109)+ step 6(ccr 瘦身c4102813c)+ steps 7-8(orphan_blocks + state_fieldsa03c666b1)+ CI 门修复(01b69f297、c28489e6c:unused-export/架构图门、null-orig 契约、孤儿块经合并视图读、orig_hash FK 索引)。每步独立绿。kernel-change: none(host 侧存储平面;kernel/ 零改动,身份不变量由既有测试族钉住)。