diff --git a/docs/proposals/snapshot-compression-foundation/README.md b/docs/proposals/snapshot-compression-foundation/README.md new file mode 100644 index 0000000000..4829dacd65 --- /dev/null +++ b/docs/proposals/snapshot-compression-foundation/README.md @@ -0,0 +1,363 @@ +# Proposal: Snapshot-Table Compression Foundation (six-stage fixed chain, single storage truth) + +> **Status: design proposal, awaiting owner ruling. No code implemented.** +> **Baseline:** origin/master (written 2026-11-07; storage schema v11; #5854 has fully reverted the v6 line from master). +> **Origin thread:** [#5643](https://github.com/ranxianglei/billion-context/issues/5643) (incl. the re-positioning comment). +> **Companion doc:** [中文版](./README.zh-CN.md) (fuller wording where the two diverge). + +--- + +## 0. TL;DR + +Replace today's parallel state sources (process memory + kstate JSON + table spread) with **one snapshot table set as the single storage truth**. Every model request walks the same fixed chain end to end: + +**raw bytes → protocol conversion (canonical cores) → rewrite/compression → apply to snapshot tables → egress mapping → outbound** + +The compression kernel (fold judgment, summaries, prompts, ref/msgid contracts) stays **byte-for-byte untouched**. Sessions whose host supplies both a session id and per-element msgids run entirely on the new chain; hosts without msgids degrade to order-then-hash identity and write into the same tables. Any mismatch anywhere sends that turn back to today's legacy full-resend path (degrade, never guess). Rollback = DROP the four new tables + flip the flag off — byte-identical return to today. + +Objective ordering is unchanged: **stability / extensibility / generality first, performance is a byproduct** — ask "is the flow correct", not "is it fast". + +--- + +## 1. Background & positioning + +### 1.1 History + +- **#5643** started as the implementation contract for "fast-forward watermark v6": a `(sess_raw, msgid)` composite-identity incremental fast lane for plugin hosts. +- The entire v6 line (fastpath + render-snapshot plane) was reverted out of master via **[#5854](https://github.com/ranxianglei/billion-context/pull/5854)** (clean revert, merged). +- The owner's re-positioning comment on #5643 (2026-11-07) fixed the new direction: this line is **not an incremental lane**, and not a sequel of v5 watermark — it is a **snapshot-table-based compression foundation**: storage-centric, kernel unchanged, with one fixed chain deciding what gets rewritten, stored, and rendered. +- [#5855](https://github.com/ranxianglei/billion-context/pull/5855) (transaction-skeleton re-land) is the **engineering substrate** of this line, not its end state. It was still open (conflicts unresolved) when this proposal was written — the first timing dependency. + +### 1.2 What this line is NOT (drift guard) + +| Easily conflated direction | Why it's not that | +|---|---| +| The incremental plugin-turn lane | That lane optimizes REQUEST SHAPE (delta-only transfer); here the server still receives full history each turn — it just stops doing indiscriminate parse/store over all of it | +| A sequel of the v5 watermark fastpath | v5 attacks PARSE COST (stamp handshake + tail-window parsing); this proposal fixes the STATE TRUTH problem; v5 keeps serving non-cohort traffic | +| Kernel swap / fold-algorithm change | Kernel zero-change is a hard constraint: fold judgment, summaries, cadence, id contracts stay as-is | +| A "faster" perf feature | Perf wins (stable prefix cache, lower memory) are byproducts of a correct flow, never the design criterion | + +### 1.3 Carried baseline rulings + +All standing rulings from the #5643 body remain in force without restatement here: objective ordering, the fixed chain, the two verification points (kernel-internal state + post-rewrite canonical form), append-only row keys, evidence-addressed diffing only (no fuzzy similarity), the four-piece protection (tripwire / S4 per-row checksums / witness requirement / audit sampling), pure-ignore of client-side compaction, invariants P5–P9. Conflicts, if any, must be registered explicitly in the ruling table (§14). + +--- + +## 2. Current architecture & three structural problems + +![Current architecture](./assets/arch-current.svg) (bitmap fallback: [arch-current.png](./assets/arch-current.png)) + +Today a model request passes through `prepare-*` adapters → `lane-decisions` branching (legacy full resend / incremental plugin-turn / v5 watermark 5% rollout / D-lane snapshot render ≈0% rollout) → kernel → `coreTo*` egress mapping. State lives in three places: process memory (lastMessages ASTs, resident core arrays), the unified SQLite v11 table family, and kstate JSON (hydrated through `mergeState()`). Three structural problems: + +1. **State lives in three places.** kstate JSON and the tables form a dual-write fork point; the direct producer of the #5403 OOM class is lastMessages pinned whole; restart depends on hydration assembly — assemble wrong and you either degrade or brick the session. The line we just reverted had exactly this parallel-truth disease at its root. +2. **Identity never lands on disk.** Host `(sess_raw, msgid)` pairs are stripped at ingest and live exactly one round in-flight. The `messages.msgId` column stores bili/kernel canonical ids; no table carries the host's raw identity. "Have I seen this element before?" cannot be answered across processes or restarts (P8 requires it always answerable), so every round cold-starts re-probing. +3. **Fastpaths patch independently.** v5 / incremental lane / D-lane each own a slice of traffic; cache stability is layered patches stacked on top of each other. One layer out of step = cache poisoning or full fallback. There is no single storage truth underneath them. + +--- + +## 3. Target architecture overview + +The full view is **pixel-aligned one-to-one with the §2 current-architecture diagram** — compare side by side to see exactly which part of the global picture changed and how big the blast radius is (grey = untouched · cyan = new stage/table · amber = role change · red dashed = fallback safety net): + +![Target architecture](./assets/arch-target.svg) (bitmap fallback: [arch-target.png](./assets/arch-target.png)) + +How the six-stage chain flows on the inside — **stage-by-stage expansion** (commit primitive / DDL chips / red-gate band R1–R4 / coexistence routing & rollout rollback): + +![Stage expansion](./assets/arch-stages.svg) (bitmap fallback: [arch-stages.png](./assets/arch-stages.png)) + +Where each piece of today's architecture lands under the new design — **component migration map** (bottom band maps the three structural pain points one-to-one to their fixes): + +![Component migration map](./assets/arch-migration.svg) (bitmap fallback: [arch-migration.png](./assets/arch-migration.png)) + +One-line duties per stage: + +| # | Stage | One line | Status | +|---|------|----------|--------| +| 0 | Ingress · identity registration | Dual-channel identity + L1 world match decides each row's provenance this turn | **NEW** | +| 1 | Protocol conversion → canonical cores | Reuse existing wire matrix; span-parse only the identity-diff NEW elements | Existing + one new regression pin | +| 2 | Rewrite / compression | Kernel untouched; emits structured events; landing timing per appendix-two | Kernel zero-change | +| 3 | Apply to snapshot tables | Single-transaction commit primitive writes the snap trio (rows/log/ckpt); ckpt meta carries kernel state + coverage map + S4 | **NEW CORE** | +| 4 | Egress mapping | Live rows ordered by pos ++ fresh-parse of tail window; dual-carrier branch | Read-side rebuild | +| 5 | Outbound | Tripwire sampling comparison before release upstream | Guard strengthened | + +Two cross-cutting bands: the **red-gate degradation band** (§7 — any doubt falls back to legacy) and **coexistence routing** (§8 — every traffic class has exactly one home). + +--- + +## 4. Stage-by-stage design + +### 4.0 Stage 0 — Ingress · identity registration (does not exist today) + +**Inputs:** the raw body plus dual-channel identity — `sess_raw` (host session id, original value) in the request header, per-element `msgids[]` in envelopes. **Collection reuses the carrier resolvers already merged in [#5842](https://github.com/ranxianglei/billion-context/pull/5842)** (shim / pi entry-walker / opencode transform hook / codex / cc / gemini candidate carriers) — no second collection system. + +**World-match decision** (sequential, first hit wins): + +1. **Echo direct-hit + spot check:** the id list the host echoes matches the server transcript positionally; adopt after sampled witness verification; +2. **Composite probe:** `(sess_raw, msgid)` hits the L1 table ⇒ continue the existing ledger identity; +3. **Mint new identity:** neither hits ⇒ mint a new ledger id, record it in L1, pay one arming round (that element enters storage but not cache decisions). + +**Cohort vs degradation boundary** (ruling I, §14): a cohort host (one that DECLARES dual-channel capability) missing ids in some round = failure → hard reject + zero side effects + feed the breaker (baseline §3.2 verbatim). Non-cohort hosts without msgids take the sticky order-then-hash degraded channel writing into the same tables (pos issued by ordinal). The mode is sticky per session; switching mid-session is one explicit resync — silent mixing forbidden. + +**L1 table** (the vehicle of ruling G): `l1_world(sess_raw PK, ledger_id, …)`, queryable globally and across restarts; original values never truncated (P9); sig/fingerprint go to warn diagnostics only, never into judgment (P8/P9). + +### 4.1 Stage 1 — Protocol conversion → canonical cores (reuse) + +- The wire matrix `{openai,anthropic,responses,google}ToCore ↔ coreToX` and cross-wire stable canonical ids (#3642) are already on master — zero rebuild. +- **Span parsing** covers only the NEW element set decided in stage 0 — these can sit at ANY body position (a client-compaction summary may appear at pos 0). Dirty implementation detail: the codec's ClusterCounter numbers `_N` per call, so parsing a scattered subset must reproduce the numbering context of a WHOLE-ARRAY call, or egress bytes drift = prefix cache breaks. **A byte-level regression pin for scattered-subset encoding is a P2 entry condition** (contiguous tail slices were proven feasible under v5; scattered subsets are the new case). +- Media cells (images etc.) reference the existing content CAS + `message_media`; base64 never enters rows. +- P5: exactly two legal entry points (fresh session / new-identity arming round); full re-parsing forbidden elsewhere. + +### 4.2 Stage 2 — Rewrite / compression (kernel zero-change) + +acp-kernel stays one line untouched: fold judgment (projectUnit shared projection), the four prompts, nudge flat-50K cadence, the id-never-reused contract. Its external output is defined here as a **structured event stream**: + +``` +cast first-time row minting (fresh session load) +append new rows (turn tail additions) +update in-row shape change (e.g. tool result backfill) +delete coverage/inactivation (active=0, never physically removed) +compact fold compaction (block absorbs covered span, meta updates coverage map) +``` + +Addressing rule follows baseline: events locate affected rows ONLY by kernel pipeline evidence (block coverage spans / ledgerSeq / frame boundaries) — no fuzzy similarity matching of any kind; if nothing lines up, the turn produces no events and runs legacy wholesale (§7 R2). + +**Landing timing (per #5643 appendix two):** first scene = the plugin endpoint `POST /__bili/plugin/tool` (src/server/admin.ts:797) completes events→table mutation **synchronously, in a single transaction, inside that request**; the next model turn only reconciles. Hosts bypassing the endpoint (dsh native compaction and kin) feed the SAME table-mutation routine via checkpoint-frame signature + rebase detection. Between endpoint and landed write the plane carries a `structural-pending` mark; losing the mark across a process restart = conservatively dead (fail-safe direction, §7 R4). + +### 4.3 Stage 3 — Apply to snapshot tables (the body of this proposal) + +**Commit primitive = one transaction** with three steps: + +``` +upsert snap_rows (rows addressed by evidence) + → append snap_log (seq strictly > max(log.max_seq, ckpt.seq)) + → (optional) compact snap_ckpt (absorb log payloads, update meta) +any step throws ⇒ abandon the whole turn, no half-state (invariant i) +``` + +**Invariant table** (draft v0.1 + review hardening, carried forward): + +| # | Invariant | +|---|-----------| +| i | Single writer per session: ALL table mutations go through this one primitive | +| ii | log.seq strictly monotonic: > max(log.max_seq, ckpt.seq); no numbering break across compactions | +| iii | S4: per-row checksum mismatch ⇒ that sid's plane is CLOSED FOREVER, whole thing falls back to the old path (red line unchanged) | +| iv | Rows are never physically deleted: folds flip active; pre/post forms preserved for decompress | +| v | post_json is always the "right-of-rewrite canonical form" — wire bytes may never be cached | + +**ckpt.meta_json triple** (where rulings F+C converge): (1) serialized kernel conversation state (CompressionState: blocks/refs/deadRefs/nudge/tokenSnapshot/…) + its own checksum — the kernel has NO deterministic "event stream → state" rebuild entry, so serialization-with-checkpoint is the only option that both keeps the kernel untouched and satisfies P7 (memory is just a cache); (2) the **coverage map** (summary pos → covered pos ranges) — delete/compact payloads get absorbed into the checkpoint, and after compaction this explicit map is decompress/GC's sole basis; (3) the S4 aggregate checksum. + +**Inactive-row lifecycle** (ruling B recommendation): inactive rows persist until their covering block itself is archived/deleted, then physical cleanup rides along; against both "never clean" (#3752 unbounded growth is a p1 red line) and global TTL (avoids extra alignment proofs between GC windows and dereference windows). + +### 4.4 Stage 4 — Egress mapping + +`mapEgress` = live rows ordered by pos, their cores, ++ freshly parsed cores of the tail window `[K..T)` this turn, then `coreToX` into the target protocol family. Prefix bytes stable across turns ⇒ prefix-cache friendly — this IS the objective function, structurally guaranteed rather than coincidental. + +- **Dual-carrier branch** (sticky per session; BOTH modes must be reasoned together — classic #377-class bug in this repo): pluginMode suppresses tool+nudge injection (the agent executes compression locally); proxy mode revoices acp_summary as a user message (systemToUser, keeping the head system message byte-stable). +- **Transparent changes** (NOT red conditions): switching model across protocol families, maxDim change, toolArmed change — they only vary mapping inputs, since snapshots store only the right-of-rewrite canonical form. +- Desync handling: join-chain broken / fails to align ⇒ the whole turn runs the legacy full path and re-stamps (client converges within one round). + +### 4.5 Stage 5 — Outbound + +**Tripwire runtime form** (ruling J recommendation): low-ratio sampling compares egress bytes word-for-word against the legacy full-path artifact (modulo time attributes). Match ⇒ release; mismatch ⇒ that session is immediately dropped back to legacy + alert. This final guard protects against "feeding stale prefixes out of a drifted state" — cache poisoning. + +**Protocol hygiene:** pf-* / envelope-style handshake headers are protocol headers, not hop headers — strip before forwarding upstream (UPSTREAM_HOP_HEADERS whitelist discipline); any NEW handshake header introduced by this proposal joins the same whitelist management. A/B byte pins stay in place throughout rollout. + +--- + +## 5. Storage schema (proposal skeleton; final DDL lands in the P0 contract) + +Version-stacking rule: once #5855 merges and lands its own schema version (draft assumes v12), this plan bumps one more (v13+). Old-library upgrade is purely additive (`CREATE ... IF NOT EXISTS`), no data migration. DDL skeleton (exact column types finalized in P0 contract review): + +```sql +-- L1 identity thin table (vehicle of ruling G; restores baseline §4.1.1's two columns as a dedicated table) +CREATE TABLE IF NOT EXISTS l1_world( + sess_raw TEXT NOT NULL, -- host session id, ORIGINAL value, never truncated (P9) + ledger_id TEXT NOT NULL, -- bili canonical sid (existing key space, ruling D) + witness_probe TEXT, -- echo spot-check witness sample (diagnostics stream only) + issued_at INTEGER NOT NULL, + armed_at INTEGER, -- arming-round mark for freshly minted identities + PRIMARY KEY (sess_raw) +); +CREATE INDEX IF NOT EXISTS idx_l1_ledger ON l1_world(ledger_id); + +-- Row table: the transcript proper +CREATE TABLE IF NOT EXISTS snap_rows( + sid TEXT NOT NULL, + pos INTEGER NOT NULL, -- ledger absolute position, append-only, never reused + core_ids TEXT NOT NULL, -- canonical-core element id list (json text) + pre_json TEXT, -- pre-rewrite form reference (CAS digest); ruling A② defaults to reference, not inline + post_json TEXT NOT NULL, -- post-rewrite canonical form (the ONLY form allowed to be cached) + tokens_est INTEGER, -- estimated tokens (ruling L: growth data source for judgment + billing share) + active INTEGER NOT NULL DEFAULT 1, -- 0 = covered/inactive (kept for decompress; lifecycle per §4.3) + cover_ref TEXT, -- id of the summary block covering it (lifecycle anchor for physical cleanup) + updated_at INTEGER NOT NULL, + PRIMARY KEY (sid, pos) +); + +-- Event log (LSM log layer) +CREATE TABLE IF NOT EXISTS snap_log( + sid TEXT NOT NULL, + seq INTEGER NOT NULL, -- strictly > max(log.max_seq, ckpt.seq) + kind TEXT NOT NULL, -- cast|append|update|delete|compact + payload TEXT NOT NULL, + created_at INTEGER NOT NULL, + PRIMARY KEY (sid, seq) +); + +-- Checkpoint (LSM meta layer; absorbs log payloads AND explicitly retains the coverage map) +CREATE TABLE IF NOT EXISTS snap_ckpt( + sid TEXT NOT NULL PRIMARY KEY, + ckpt_seq INTEGER NOT NULL, + meta_json TEXT NOT NULL, -- serialized kernel state + coverage map + own checksum + checksum TEXT NOT NULL, -- S4 aggregate checksum + updated_at INTEGER NOT NULL +); +``` + +**Write-model assumption** (ruling H item): default is the single-process assumption (WAL serializes cross-process writes naturally). If multi-instance concurrent writes are wanted, an in-memory-view invalidation protocol must ship with it (who invalidates resident core arrays / metadata caches on commit) — P0 contract picks one, definitively. + +--- + +## 6. Read paths & consumer-switching ledger + +**Hydration dual path:** green path (new tables readable AND S4 passes) rebuilds the read view from snap_rows + meta; red path (flag off / no active block / tables unreadable / checksum fail) falls back to the kstate-old path verbatim. Switching is per-session and one-directional; any step out of alignment sends the whole turn back to legacy — same discipline as D-lane #3756. + +Consumers annotated individually (P0 lists the complete inventory; this is the mainline): + +| Consumer | Disposition | +|----------|-------------| +| decompress (restore/decompress tool path) | P3 switch to new-table pre/post dual forms + coverage map; missing pre bytes MUST surface through an explicit audit diagnostics exit — never silent empty references | +| export | P3 switch; honest-output discipline unchanged (#155: a 0-block session may never claim "original conversation") | +| billing | shares the tokens_est column; switching read source does not change billing semantics | +| Web UI (sessions panel / acp_status uncompressed view) | acp_status judgment face (KDD#12 shared projection) input source points at new-table rows; panel gains table-health metrics (§10) | +| kv_store namespaces / notice store / persona fork (`\|sub:`) | Untouched (independent mechanisms; the fork records its suffixed id into the same lane) | +| CCR (ccr_entries) | Not touched during P0–P2; P3 evaluates merge vs long-term coexistence separately | + +--- + +## 7. Red-gate enumeration (hit ANY ⇒ the turn/session falls back wholesale to the legacy full path and re-stamps) + +| # | Red condition | Consequence | +|---|---------------|-------------| +| R1 | S4 / checksum corrupted | that sid's plane CLOSED FOREVER, whole fallback to the old path (substrate corruption unrecoverable; red line unchanged) | +| R2 | Egress join chain broken (core_ids ↔ rows fail to align; chain walk-back fails) | no events produced this turn; legacy wholesale | +| R3 | Cohost identity gate fault (cohort host fails to deliver dual-channel ids) | hard reject + zero side effects + feed breaker (baseline §3.2 verbatim) | +| R4 | structural-pending unconsumed (incl. restart-lost mark) | conservatively dead (fail-safe direction) | + +**NOT red conditions:** model switch across protocol families, maxDim change, toolArmed change, client-side compaction traces (purely ignored — increments only). These boundaries become decision functions in the lane-decisions matrix, one positive+negative row pair per combination (#3617 discipline). + +--- + +## 8. Coexistence routing with existing lanes (every traffic class has exactly one home) + +| Traffic | Home | Notes | +|---------|------|-------| +| Cohort hosts (dual channel) full-resend sessions | **New foundation** | the primary served population | +| Non-cohort (no msgid) traffic | v5 watermark keeps serving | same tables, different lane; independent flag bits; the order-then-hash degraded writes use the same tables | +| Anthropic-wire divert turns | incremental plugin-turn lane keeps its job | dsh shim picks one per turn; no double processing | +| Endpoint-bypassing hosts (dsh native compaction et al.) | stage-2's SAME event routine (checkpoint-frame signature + rebase detection feed-in) | see §4.2 timing | +| #3756 D-lane | **Yields/retires** on lanes where the new foundation is live (flag mutual exclusion) | pending ruling K; retirement decision lands in P3 | + +lane-decisions matrix: one row per combination, positive+negative pairs ("triggers when it should; does NOT trigger when it should not"); added rows discoverable. + +--- + +## 9. Rollout & config surface (owner-gated; key names finalized in the P0 config-surface section) + +- **Two-level nested flags:** master flag ⊃ lane flags (a lane flag means anything only when master is on), config-file keys resolved through `src/knobs.ts` (env > file > default); **no new env knobs** (scarce-surface discipline #2030). Exact key names declared in the P0 PR's config-surface section before sign-off. +- **Cohort roster:** config-level declaration (which carriers/hosts claim dual-channel capability); initial value TBD (open item D4: verify the six candidate carriers one by one). +- **Web config page** tri-state quick switches shipped in lockstep; i18n en/zh pairs; registry-test dead-key guard rows; website/config-reference yaml seeds where a knob ships. +- Default OFF, bucketed per session (an independent salted bucket), P2 starts as single-lane gray rollout. + +--- + +## 10. Observability surface + +- **Decision log:** unified vocabulary (hit / miss / red: / degrade / resync / arm …), every line grep-able and aggregatable; during rollout it IS the confidence evidence. +- **`/__bili/status`:** attribution field extended with foundation dimensions (which chain serves this session, plane health). +- **Panel health metrics:** snap table row counts / log depth / ckpt lag (seq distance) / GC waterline / tripwire sampling agreement rate — sessions panel or a dedicated card. +- **acp_status uncompressed view:** judgment input source repointed to new-table rows (KDD#12 shared-projection apply/recommend/status consistency re-pinned on the new tables). + +--- + +## 11. Test matrix + +Phase test skeleton (positive+negative pairing applies throughout): + +| # | Test | Phase | +|---|------|-------| +| T1 | Upgrade additivity pin: pre/post-upgrade existing query behavior byte-identical; DROP-four-new-tables rollback drill = byte-identical return to today | P0 | +| T2 | Event transaction unit: commit primitive atomicity; seq monotonic across compaction; throw leaves no half-state | P0 | +| T3 | B1 trap pin × 3 consecutive restarts oracle comparison (hydrated snapshot must carry the active world origin o; frontier movable only by arm/reset) | P1 | +| T4 | A/B byte-level: on N-turn real corpora, new-chain egress bytes == legacy full-path artifact (modulo time attr) | P1–P3, always on | +| T5 | Scattered-NEW encoding byte-equivalence pin (ClusterCounter reproduction) | P2 entry | +| T6 | deleteSession cascade incl. l1_world — no orphans | P0/P1 | +| T7 | Coverage map survives compaction: decompress/GC basis still queryable after compact | P1 | +| T8 | Post-GC decompress honest-failure exit (missing pre ≠ silent empty reference) | P3 | +| T9 | Long-session volume growth curve model (row count / log depth / ckpt size vs the #3752 red line), long-run acceptance | P1 | +| T10 | Red-gate matrix: R1–R4 each with a "triggers-and-falls-back" + "must-NOT-trigger" pair row | per phase, alongside modules | +| T11 | Real-client e2e (pi / dsh × sglang) full chain | P2 entry | +| T12 | Dual-mode (pluginMode/proxyMode) carrier branch double verification (§4.4) | P2 | + +CI: docker gate + solver-free; hermetic; ports via listen(0) (#360 discipline). + +--- + +## 12. Phased delivery (each phase = its own PR; human gates marked) + +| Phase | Scope | Behavior surface | Acceptance | +|-------|-------|------------------|------------| +| **P0** | DDL four-piece + commit primitive + upgrade pin + rollback drill + event transaction unit tests | **ZERO behavior change** (foundation laying only) | T1/T2/T6; write-model assumption (H) and key naming (config-surface section) finalized here — human gate ① | +| **P1** | Shadow write: new tables populated in parallel with the old path; read side completely untouched | Zero behavior surface | T3/T4(shadow compare)/T7/T9 long-run curves; volume model passes red-line check — human gate ② (critical trust-rebuild period) | +| **P2** | Single-lane hydrate-egress cutover gray rollout (first cohort member; suggest pi or dsh) | Gray sessions read egress from new tables | T5/T10/T11/T12 + A/B pins always on + tripwire sampling present; one-click exit switch | +| **P3** | decompress/export/UI/billing switched one consumer at a time; CCR/persona disposition; #3756 retirement decision; cohort roster expansion | Per-consumer switches | T8 + full matrix rerun; each phase its own PR | + +**Ruling E recommendation stands:** this effort scopes to P0+P1, hands off for acceptance; P2 concentrates the byte-parity risk and ships as its own PR once gray-rollout infrastructure is ready. + +--- + +## 13. Prerequisites & sequencing + +1. **#5855 merges FIRST** (human merge): it is the engineering substrate of the transaction skeleton AND the schema-version layer beneath ours. While it sits open/dirty, P0 DDL cannot stack versions cleanly. +2. #5842 (identity collection) ✓ already on master — stage 0 reuses it directly; nothing to wait for. +3. #5854 (revert) ✓ merged — confirms zero v6 symbol residue on master. +4. Rulings closed (§14): F/G/B decide the table-shape skeleton — decide those first; the rest ride with the P0 PR. + +--- + +## 14. Pending owner rulings + +Already discussed online (A–E, draft-v0.1 numbering) + gap-analysis additions (F–L) + carryovers (D3–D5): + +| # | Item | Recommendation | Blocks | +|---|------|----------------|--------| +| A | In-row form | ② store post only, pre references existing CAS; addendum: pre bytes referenced by a LIVE row must outlive that row | P0 DDL | +| **B** | Inactive-row retention | compaction cleanup bound to covering-block lifecycle; against both never-clean (#3752 red line) and global TTL | P0 DDL / GC design | +| C | Summary placement | ① occupy covered-span head pos in place + ckpt meta explicitly retains the coverage map | P0 DDL | +| D | Session key space | reuse existing sid | P0 DDL | +| E | Scope this cycle | hand off at P0+P1 for acceptance; P2 separate PR with rollout | Scheduling | +| **F** | Kernel state residence | ckpt.meta_json carries serialized CompressionState + checksum (kernel untouched, O(1) hydration) | P0/P1 core | +| **G** | L1 identity thin table | restore persistent (sess_raw, ledger_id) per baseline §4.1.1 (dedicated thin table + index), reusing #5842 collection | P0 DDL | +| **H** | Write model | single-process assumption (WAL serializes) OR multi-process + in-memory-view invalidation protocol — pick one, definitive, in P0 | P0 | +| **I** | Hard gate vs degradation boundary | cohort-claimed host: fault reject + zero side effects + breaker; non-cohort: order-then-hash same-table degraded writes, sticky per session, mode switch = resync | P0/P2 routing | +| **J** | Tripwire | runtime low-ratio sampling + immediate legacy drop + alert on mismatch | P2 | +| **K** | Coexistence with #3756 / v5 | #3756 yields/retires (flag mutual exclusion) on new-foundation lanes; v5 continues serving non-cohort | P2/P3 | +| **L** | In-row token accounting column | tokens_est in-table: growth data source for judgment + billing share | P0 DDL | +| D3 | Fault-response shape | HTTP code / error body / client retry contract (peer-side view of R3 hard reject) | P2 | +| D4 | Initial cohort roster | verify the six candidate carriers (shim/pi/opencode/codex/cc/gemini) one by one for carrier stability | P0 config initial value | +| D5 | Re-numbering semantics | same content, different id: count as new duplicate pair or pair-level dedup? | P0 semantics | + +--- + +## 15. Risks & honest boundaries + +- **"Element-count-preserving prefix byte rewrites are undetectable"** (carried cost clause from baseline): the price is a trust-model change that skips per-item re-verification — human review required, auto-merge forbidden (same gate tier as KDD §7.4 items). +- **Once a multimodal tail enters the cache, that history segment permanently degrades to legacy full turns** (carried KDD#15 honest boundary ②): degrade rather than deviate. +- **Marker-strip counts inherit with the seed ledger**; mid-history regions are never rescanned (boundary ③ above). +- The final arbiter of every "invariant" in this document is the runtime tripwire + S4 sampling, not paper argument — which is precisely why the P1 shadow phase exists: to accumulate that empirical evidence before any read-side cutover. + +--- + +## 16. Reference numbers + +#5643 (origin + re-positioning) · #5854 (full revert, merged) · #5855 (substrate re-land, open) · #5842 (identity wiring, merged) · #3756 (D-lane snapshot render; disposition pending ruling K) · #3752 (unbounded storage growth p1 red line) · #5403 (OOM: lastMessages pinning) · #2432 (client-compaction death spiral; source of the pure-ignore policy) · #3236 (shared projection judgment) · #3642 (cross-wire stable canonical ids) · #3617 (gate bidirectional assertions) · #1104/#1106/#1107 (fingerprint permitlist discipline) · #4491/#4834 (msgid identity hierarchy) · #5539 (schema v11 FK subindex) · #155 (honest output) · #360 (listen(0) port discipline) · MESSAGE-IDENTITY.md (identity doctrine) · #5643 re-positioning comment: diff --git a/docs/proposals/snapshot-compression-foundation/README.zh-CN.md b/docs/proposals/snapshot-compression-foundation/README.zh-CN.md new file mode 100644 index 0000000000..c1c44a6836 --- /dev/null +++ b/docs/proposals/snapshot-compression-foundation/README.zh-CN.md @@ -0,0 +1,363 @@ +# 提案:快照表式压缩底座(六段固定链,单一存储真源) + +> **状态:设计提案,待 owner 裁定后开工。** 未实施任何代码。 +> **基线:** origin/master(写于 2026-11-07;存储 schema v11,#5854 已把 v6 整线撤出 master)。 +> **来源讨论:** [#5643](https://github.com/ranxianglei/billion-context/issues/5643)(含定位纠正评论)。 +> **姊妹文档:** [English version](./README.md)。 + +--- + +## 0. 摘要(TL;DR) + +本提案把上下文压缩的状态底座从「进程内存 + kstate JSON + 多张表」的平行真源格局,换成**一套快照表作为唯一存储真源**。每轮请求固定走完同一条链: + +**原始字节 → 协议转换(通用核) → 改写(压缩) → 快照表应用 → 出站映射 → 出站** + +压缩内核(折判定、摘要、prompts、ref/msgid 契约)**零改动**。有宿主 session+msgid 双信道的会话全部走新链路;没有 msgid 的宿主按「顺序 > hash」降级身份写同一套表。任何环节失步,该轮整体回落今天的 legacy 全量路径(降级,不猜)。回滚 = DROP 四张新表 + 关旗,逐字节回到今天。 + +目标函数排序不变:**稳定性 / 扩展性 / 通用性优先,性能是副产品**——先问流转对不对,不问快不快。 + +--- + +## 1. 背景与定位 + +### 1.1 来龙去脉 + +- **#5643** 最初是「快进水位 v6」实现契约:用 `(sess_raw, msgid)` 复合身份给插件车道做增量快路。 +- 整条 v6 线(快路径 + render snapshot 平面)经 **[#5854](https://github.com/ranxianglei/billion-context/pull/5854)** 完整撤出 master(干净 revert)。 +- 撤回后的定位纠正(#5643 评论区,2026-11-07)把方向定死了:这条线**不是增量车道,也不是 v5 watermark 的续作**,而是**基于快照表的压缩底座**——以新存储为主,内核不动,让「改什么、存什么、渲染什么」全部由一条固定链决定。 +- [#5855](https://github.com/ranxianglei/billion-context/pull/5855)(事务骨架 re-land)是本线的**工程基底**,不是终态。截至本提案写作时它仍 open(冲突未解)——这是本提案的第一时序依赖。 + +### 1.2 这条线不是什么(防止再次漂移) + +| 容易误认的方向 | 为什么不是 | +|---|---| +| 增量车道(plugin-turn divert) | 那条是「只传新增 delta」的请求形状优化;本提案每轮仍收全量重发,只是服务端不再对全量做无差别解析/存储 | +| v5 watermark 快路径的续作 | v5 解决的是「解析成本」(戳握手 + 尾窗解析);本提案解决的是「状态真源」,v5 继续服务非 cohort 流量 | +| 换内核/改折叠算法 | 内核零改动是硬约束;折叠判定、摘要、节拍、id 契约原样保留 | +| 「快一点」的性能特性 | 性能收益(前缀缓存稳定、内存下降)是流转正确的副产品,不作为设计判据 | + +### 1.3 保留的基准裁定(继承自 #5643 正文,全文有效) + +目标函数排序、固定链、两处校验点(内核内部状态 + 改写后核形态)、行键 append-only、diff 只按内核管道证据寻址(禁模糊相似)、四件套保护(tripwire / S4 逐行校验和 / 见证要求 / 审计抽验)、客户端压缩纯忽略、P5–P9 五组不变量,全部沿用,不在本文重复展开;与本文冲突处以本节为准并在裁定表(§14)显式登记。 + +--- + +## 2. 现状架构与三个结构性问题 + +![现状架构](./assets/arch-current.svg)(位图备选:[arch-current.png](./assets/arch-current.png)) + +今天的一条模型请求从入站到出站要经过:`prepare-*` 适配器 → `lane-decisions` 分轨(legacy 全量 / 增量 plugin-turn / v5 watermark 5%灰度 / D-lane 快照渲染 ≈0%灰度)→ 内核 → `coreTo*` 出站映射。状态住在三个地方:进程内存(lastMessages AST、常驻核数组)、统一 SQLite v11 表族、kstate JSON(经 `mergeState()` 水合)。三个结构性问题: + +1. **状态三处住(痛点①)**——kstate JSON 与表是双通道分叉点;#5403 OOM 的直接生产者就是 lastMessages 整包驻留;重启靠水合拼装,拼错即降级或烧会话。刚被撤回的那条线的根病正是这种平行真源。 +2. **身份不落盘(痛点②)**——宿主的 `(sess_raw, msgid)` 对在 ingest 时剥掉,只活在 in-flight 信封里一轮。`messages.msgId` 列存的是 bili/内核自己的规范 id,宿主的原始身份没有任何表承载。「这个元素见过没?」跨进程、跨重启无法回答(P8 要求恒可答),于是每轮都冷启动重新探测。 +3. **快路径各自打补丁(痛点③)**——v5 / 增量车道 / D-lane 各管一段流量,缓存稳定性靠多层补丁堆叠;任何一层失步就是缓存投毒或整体回退。没有一个单一存储真源兜底。 + +--- + +## 3. 目标架构总览 + +全貌与 §2 现状架构图**逐位对齐**,直接对照即可看出改的是全局哪一块、影响范围多大(灰 = 零改动 · 青 = 新增段/表 · 琥珀 = 角色变更 · 红虚线 = 兜底安全网): + +![目标架构](./assets/arch-target.svg)(位图备选:[arch-target.png](./assets/arch-target.png)) + +六段固定链自身的内部流转 —— **分阶段展开**(提交原语 / DDL chip / 降级红门 R1–R4 / 共存路由与回滚灰度): + +![分阶段展开](./assets/arch-stages.svg)(位图备选:[arch-stages.png](./assets/arch-stages.png)) + +现状图里的每一块在新架构下变成什么样子 —— **组件迁移总览**(底部为三个结构性痛点的一一对应解决锚点): + +![组件迁移总览](./assets/arch-migration.svg)(位图备选:[arch-migration.png](./assets/arch-migration.png)) + +六个阶段的职责一句话版: + +| # | 阶段 | 一句话 | 状态 | +|---|------|--------|------| +| 0 | 入站·身份入册 | 双信道身份 + L1 世界匹配,决定这轮每行的出身 | **新增** | +| 1 | 协议转换→通用核 | wire 矩阵现成复用;只对身份差集 NEW 元素做 span 解析 | 现成 + 一个新回归钉 | +| 2 | 改写/压缩 | 内核一行不改,产出结构化事件;落表时序按附录二 | 内核零改动 | +| 3 | 快照表应用 | 单事务提交原语写入 snap 三件套(rows/log/ckpt);ckpt meta 携带内核态+覆盖映射+S4 | **新增核心** | +| 4 | 出站映射 | 存活行按 pos 排序 ++ 尾窗新解析;双 carrier 分支 | 读侧重建 | +| 5 | 出站 | tripwire 抽样比对后放行上游 | 守卫强化 | + +横切两条:**红门降级带**(§7,任何疑点回落 legacy)与**共存路由**(§8,每种流量只有一个家)。 + +--- + +## 4. 六段固定链逐段设计 + +### 4.0 第 0 段:入站 · 身份入册(现状中不存在的一段) + +**输入**:原始字节体 + 双信道身份——请求头里的 `sess_raw`(宿主会话 id 原值)+ 逐元素 `msgid[]` 信封。**采集直接消费 [#5842](https://github.com/ranxianglei/billion-context/pull/5842) 已合入的载体 resolver 输出**(shim / pi entry-walker / opencode transform 钩子 / codex / cc / gemini 六候选载体),不建第二套采集系统。 + +**世界匹配裁决**(顺序执行,第一个命中即止): + +1. **echo 直中 + 抽验**:宿主回带的 id 列表与服务端正史逐位吻合,抽样见证核对通过后直接采用; +2. **复合探测**:`(sess_raw, msgid)` 在 L1 表中探测命中 ⇒ 沿用既有账本身份; +3. **发放新身份**:都没命中 ⇒ 发放新账本 id 并记入 L1,付一轮武装(本轮该元素只进不入缓存决策)。 + +**cohort vs 降级的边界**(裁定 I,见 §14):声称具备双信道的 cohort 宿主某轮收不齐 id = 故障,硬拒绝 + 零副作用 + 喂 breaker(baseline §3.2 原样);天生没有 msgid 的非 cohort 宿主走「顺序 > hash」粘滞降级通道,写同一套表(pos 由序位发号);模式每会话粘滞,中途换模式 = 一次显式 resync,禁止静默混用。 + +**L1 表**(G 裁定的承载体):`l1_world(sess_raw PK, ledger_id, …)`,全局可跨重启查询;原值禁截断(P9),sig/fingerprint 只进 warn 诊断流永不入判决(P8/P9)。 + +### 4.1 第 1 段:协议转换 → 通用核(现成复用) + +- wire 矩阵 `{openai,anthropic,responses,google}ToCore ↔ coreToX` 与跨线稳定规范 id(#3642)均在 master,零重建。 +- **span 解析**:只对第 0 段判定的 NEW 元素集合做解析——它们可以散布在 body 任意体位(客户端压缩后摘要可能出现在 pos 0)。这是一个脏实现点:codec 的 ClusterCounter 按单次调用内次序编号 `_N`,解析散布子集必须复现「整数组调用」的编号上下文,否则出站字节漂移 = 前缀缓存碎掉。**散布子集编码的字节级回归钉是 P2 准入条件**(连续尾切片情形 v5 已证可行,散布子集是新情形)。 +- 媒体格(图片等)走既有 content CAS + `message_media` 引用,base64 不进行表。 +- P5:合法入口恰两个(全新会话 / 新身份发放轮),其余轮次禁止全量重解析。 + +### 4.2 第 2 段:改写 / 压缩(内核零改动) + +acp-kernel 保持一行不改:折叠判定(projectUnit 共享投影)、四条 prompts、nudge flat-50K 节拍、ref/msgid 永不复用契约。它对外的输出在本设计中定义为**结构化事件流**: + +``` +cast 首次铸入行(新会话首载) +append 新增行(新回合尾部) +update 行内形态变更(如 tool result 回填) +delete 覆盖/失效(active=0,不物理删) +compact 折叠压实(块吸收覆盖段,meta 更新覆盖映射) +``` + +事件寻址规则沿用 baseline:**只按内核管道证据**(block 覆盖区间 / ledgerSeq / 帧边界)定位受影响行,禁止任何模糊相似匹配;对不上 ⇒ 该轮不产事件、整轮走 legacy(§8 R2)。 + +**落表时序(以 #5643 附录二为准)**:第一现场 = 插件端点 `POST /__bili/plugin/tool`(src/server/admin.ts:797)在**当次请求内同步单事务**完成事件→表修改;下一轮模型请求只做核对。绕过端点的宿主(dsh native compaction 一类)由 checkpoint 帧签名 + rebase 检测喂**同一个**表修改例程兜底。端点到落表之间平面挂 `structural-pending` 标记;进程重启丢失标记 = 保守判死(fail-safe 方向,§8 R4)。 + +### 4.3 第 3 段:快照表应用(本提案主体) + +**提交原语 = 单事务**,内含三步: + +``` +upsert snap_rows(按证据寻址的行) + → 追加 snap_log(seq 严格 > max(log.max_seq, ckpt_seq)) + → (可选)折 snap_ckpt(吸收 log 载荷,更新 meta) +任一步抛错 ⇒ 整轮放弃,不留半态(不变量 i) +``` + +**不变量表**(草案 v0.1 起、review 补强后沿用): + +| # | 不变量 | +|---|--------| +| i | 单会话单写者:所有表修改只经这一个提交原语 | +| ii | log.seq 严格单调:> max(log.max_seq, ckpt_seq),跨压实不断号 | +| iii | S4:逐行校验和不符 ⇒ 该 sid 平面永久关死,整体回落旧路(红线不变) | +| iv | 行不物理删除:折叠只翻 active,pre/post 双形态保留供 decompress | +| v | post_json 恒为「改写层右侧的通用核形态」——缓存永远不允许 wire 字节 | + +**ckpt.meta_json 三件套**(F+C 裁定的合流):① 内核会话态(CompressionState:blocks/refs/deadRefs/nudge/tokenSnapshot/…)序列化 + 自身校验和——内核没有确定性「事件流→状态」重建入口,序列化随 checkpoint 走是唯一既零改内核又满足 P7(内存只是缓存)的方案;② **覆盖映射**(摘要 pos → 被覆盖 pos 区间)——delete/compact 载荷会被 ckpt 吸收,压实之后 decompress/GC 的唯一依据就是这份显式映射;③ S4 聚合校验和。 + +**失效行生命周期**(B 裁定建议):inactive 行保留至其覆盖块本身被归档/删除时一并物理清理;反对永不物理清(#3752 单调无界增长是 p1 红线)也反对全局 TTL(GC 窗口和解引用窗口不用额外对齐证明)。 + +### 4.4 第 4 段:出站映射 + +`mapEgress` = 存活行按 pos 排序取核 ++ 尾窗 `[K..T)` 本轮新解析的核,再过 `coreToX` 到目标协议族。前缀跨轮字节稳定 ⇒ 前缀缓存友好(这就是目标函数,不是巧合而是结构保证)。 + +- **双 carrier 分支**(会话粘滞绑定,两模式必须同时想清——本仓库惯犯病类 #377):pluginMode 抑制 tool+nudge 注入(agent 本地执行压缩);proxy 模式 acp_summary revoice 为 user 消息(systemToUser,保护头部 system 逐字稳定)。 +- **透明变化**(不是红条件):跨协议族切模型、maxDim 变、toolArmed 变——都只动映射输入。因为快照只存改写右侧的通用核形态,族切换天然透明。 +- 失步处理:join 链断 / 对不上 ⇒ 该轮整条 legacy 全量路径并重新盖戳(客户端一轮收敛)。 + +### 4.5 第 5 段:出站 + +**tripwire 运行期形态**(J 裁定建议):低比例抽样把出站字节与 legacy 全量路径产物逐字比对(除 time attr)。一致放行;不一致 ⇒ 该会话立即摘回 legacy + 告警。最终护栏防的是「漂移状态下供出陈旧前缀 = 缓存投毒」。 + +**协议卫生**:pf-* / 信封类握手头是协议头不是 hop 头,转发上游前剥掉(UPSTREAM_HOP_HEADERS 白名单纪律);本提案引入的任何新握手头同样进白名单管理。灰度期全程 A/B 字节钉在场。 + +--- + +## 5. 存储 schema(提案骨架,P0 契约定稿) + +版本叠加规则:#5855 合并落地它自己的 schema 版本(草案假定 v12)之后,本方案在其上再升一版(v13+);老库升级纯 `CREATE ... IF NOT EXISTS` 加性变更,无数据迁移。DDL 骨架(精确列型留待 P0 契约评审): + +```sql +-- L1 身份薄表(裁定 G 的承载体;baseline §4.1.1 的两列恢复为独立表) +CREATE TABLE IF NOT EXISTS l1_world( + sess_raw TEXT NOT NULL, -- 宿主会话 id 原值,禁截断(P9) + ledger_id TEXT NOT NULL, -- bili 规范 sid(沿用现有键空间,裁定 D) + witness_probe TEXT, -- echo 抽验见证样本(仅诊断流) + issued_at INTEGER NOT NULL, + armed_at INTEGER, -- 新身份发放的武装轮标记 + PRIMARY KEY (sess_raw) +); +CREATE INDEX IF NOT EXISTS idx_l1_ledger ON l1_world(ledger_id); + +-- 行表:正史本体 +CREATE TABLE IF NOT EXISTS snap_rows( + sid TEXT NOT NULL, + pos INTEGER NOT NULL, -- ledger 绝对位置,append-only 永不复用 + core_ids TEXT NOT NULL, -- 通用核元素 id 序列(json text) + pre_json TEXT, -- 改写前形态引用(CAS digest),裁定 A② 默认引用不内嵌 + post_json TEXT NOT NULL, -- 改写后通用核形态(唯一允许被缓存的形态) + tokens_est INTEGER, -- 估算 token(裁定 L:判决面增长数据源 + 计费共用) + active INTEGER NOT NULL DEFAULT 1, -- 0=被覆盖/失效(保留供 decompress,生命周期见 §4.3) + cover_ref TEXT, -- 覆盖它的摘要块 id(物理清理的生命周期锚) + updated_at INTEGER NOT NULL, + PRIMARY KEY (sid, pos) +); + +-- 事件日志(LSM log 层) +CREATE TABLE IF NOT EXISTS snap_log( + sid TEXT NOT NULL, + seq INTEGER NOT NULL, -- 严格 > max(log.max_seq, ckpt_seq) + kind TEXT NOT NULL, -- cast|append|update|delete|compact + payload TEXT NOT NULL, + created_at INTEGER NOT NULL, + PRIMARY KEY (sid, seq) +); + +-- 检查点(LSM meta 层;吸收 log 载荷并显式保留覆盖映射) +CREATE TABLE IF NOT EXISTS snap_ckpt( + sid TEXT NOT NULL PRIMARY KEY, + ckpt_seq INTEGER NOT NULL, + meta_json TEXT NOT NULL, -- 内核态序列化 + 覆盖映射 + 自身校验和 + checksum TEXT NOT NULL, -- S4 聚合校验和 + updated_at INTEGER NOT NULL +); +``` + +**写模型假设**(H 裁定项):默认单进程假设(WAL 天然串行化跨进程写);若要多实例并发写,需附内存视图失效协议(常驻核数组/元数据缓存谁作废谁通知)——P0 契约里二选一钉死。 + +--- + +## 6. 读路径与消费方切换台账 + +**hydrate 双道**:绿道(新表可读且 S4 通过)从 snap_rows+meta 重建读视图;红道(开关关 / 无活跃块 / 表不可读 / 校验失败)原样走 kstate 旧路承接。切换按会话粒度、单向渐进,任何一道失步整轮回退 legacy——与 D-lane #3756 同纪律。 + +消费方逐个标注(P0 列全清单,此处为主干): + +| 消费方 | 处置 | +|--------|------| +| decompress(restore/decompress 工具路径) | P3 切到新表 pre/post 双形态 + 覆盖映射;pre 字节缺失时必须走审计显式诊断出口,不得静默空引用 | +| export | P3 切换;诚实输出纪律不变(#155:0-block 会话不许谎称「原始对话」) | +| 计费 | tokens_est 列共用;切读源不改变计费口径 | +| Web UI(sessions 面板 / acp_status uncompressed view) | acp_status 判决面(KDD#12 共享投影)输入源指到新表;面板新增表健康度指标(§11) | +| kv_store namespaces / notice store / persona fork(`\|sub:`) | 不触碰(独立机制,fork 记录自己后缀 id 进同一车道即可) | +| CCR(ccr_entries) | P0–P2 不碰;P3 单独评估是否并入或长期并存 | + +--- + +## 7. 降级红门枚举(命中任一 ⇒ 该轮/该会话整体回落 legacy 全量路径并重新盖戳) + +| # | 红条件 | 后果 | +|---|--------|------| +| R1 | S4 / 校验和坏 | 该 sid 平面**永久关死**,整体回落旧路(底材损坏不可恢复,红线不变) | +| R2 | 出站 join 链断(core_ids ↔ 行对不上,链回溯失败) | 该轮不产事件,整轮 legacy | +| R3 | cohort 宿主身份门故障(收不齐双信道 id) | 硬拒绝 + 零副作用 + 喂 breaker(baseline §3.2 原样) | +| R4 | structural-pending 未消费(含重启丢失标记) | 保守判死(fail-safe 方向) | + +**不是红条件的**:跨协议族切模型、maxDim 变化、toolArmed 变化、客户端压缩痕迹(纯忽略,只看增量)。把这组边界写成判定函数放进 lane-decisions 矩阵,每种组合正负成对断言(#3617 纪律)。 + +--- + +## 8. 与既有车道的共存路由(每种流量只有一个家) + +| 流量 | 归属 | 说明 | +|------|------|------| +| cohort 宿主(有双信道)全量重发会话 | **新底座** | 本提案服务的主对象 | +| 非 cohort(无 msgid)流量 | v5 watermark 快路径继续服务 | 与本底座同表不同车道,旗位独立;「顺序>hash」降级写入走同一套表 | +| anthropic wire divert 回合 | 增量 plugin-turn 车道维持原职 | dsh shim 逐回合二选一,互不双处理 | +| dsh native compaction 等绕过端点的宿主 | 第 2 段同一事件例程(checkpoint 帧签名 + rebase 检测喂入) | 见 §4.2 时序 | +| #3756 D-lane | 新底座就绪的车道**让位退役**(旗互斥) | 待裁定 K;退役判断放在 P3 | + +lane-decisions 矩阵:每个组合一行,正负成对(触发时触发、不该触发时不触发),新增行可追溯。 + +--- + +## 9. Rollout 与配置面(owner-gated,命名待 P0 config-surface 章节定稿) + +- **两级嵌套开关**:master 旗 ⊃ lane 旗(lane 旗只在 master 开时有意义),config-file key 经 `src/knobs.ts`(env > file > default)解析;**不新增 env 旋钮**(env 稀缺面纪律 #2030)。具体 key 名在 P0 PR 的 config-surface 专章申报后定稿。 +- **cohort 名单**:配置级声明(哪些 carrier/宿主声称具备双信道),初值待定(遗留 D4:shim/pi/opencode/codex/cc/gemini 六候选逐个核实)。 +- **Web「配置」页**三态快速开关同步出厂;i18n en/zh 对;registry-test 死键守护行;website/config-reference yaml 种子(若 ship 旋钮)。 +- 默认 OFF,按会话分桶(announcement-id 盐值桶之外的独立桶),P2 单车道灰度起步。 + +--- + +## 10. 观测面 + +- **决策日志**:统一词汇表(hit / miss / red: / degrade / resync / arm …),每行可 grep 可聚合;rollout 期间就是信心证据。 +- **`/__bili/status`**:attribution 字段扩展新底座维度(哪条链路在服务本会话、平面健康)。 +- **面板健康度**:snap 表行数 / log 深度 / ckpt lag(seq 落后量)/ GC 水位 / tripwire 抽验命中率,进 sessions 面板或独立小卡。 +- **acp_status uncompressed view**:判决输入源指到新表行(KDD#12 共享投影的 apply/recommend/status 三方一致性在新表上重新钉测试)。 + +--- + +## 11. 测试矩阵 + +分期测试骨架(正负成对原则全程适用): + +| # | 测试 | 期 | +|---|------|----| +| T1 | 升级纯加性钉:老库升级前后既有查询行为逐字节不变;DROP 四张新表回滚演练 = 逐字节回到今天 | P0 | +| T2 | 事件事务单元:提交原语单事务原子性;seq 单调跨压实不断号;抛错无半态 | P0 | +| T3 | B1 陷阱钉 ×3 连续重启 oracle 比对(水合快照必须携带活跃世界原点 o;frontier 只能由 arm/reset 移动) | P1 | +| T4 | A/B 字节级:N 回合真实语料下新链路出站字节 == legacy 全量产物(除 time attr) | P1–P3 常开 | +| T5 | 散布 NEW 编码字节一致钉(ClusterCounter 复现) | P2 准入 | +| T6 | deleteSession 级联连删(含 l1_world)无孤儿 | P0/P1 | +| T7 | 覆盖映射压实时存活:compact 后 decompress/GC 依据仍可查 | P1 | +| T8 | GC 后 decompress 的诚实失败出口(pre 缺失 ≠ 静默空引用) | P3 | +| T9 | 长会话体量增长曲线模型(行数/log 深度/ckpt 尺寸,对照 #3752 红线)长跑验收 | P1 | +| T10 | 红门矩阵:R1–R4 各配「触发时回落」+「不该触发时不回落」成对行 | 各期随对应模块 | +| T11 | 真实客户端 e2e(pi / dsh × sglang)全链路 | P2 准入 | +| T12 | 双模式(pluginMode/proxyMode)carrier 分支双验证(§4.4) | P2 | + +CI:docker gate + 求解器无关;hermetic,端口 listen(0)(#360 纪律)。 + +--- + +## 12. 分期交付(每期为独立 PR,人审关卡标出) + +| 期 | 范围 | 行为面 | 验收 | +|----|------|--------|------| +| **P0** | DDL 四件套 + 提交原语 + 升级钉 + 回滚演练 + 事件事务单元测试 | **零行为变化**(纯铺底) | T1/T2/T6;写模型假设(H)与 key 命名(config-surface 章)在此期定稿——人审点① | +| **P1** | 影子写:新表与旧路并行落数据,读侧完全不切 | 零行为面 | T3/T4(影子比对)/T7/T9 长跑曲线;体量模型过红线检查——人审点②(信任重建关键期) | +| **P2** | 单车道 hydrate-egress 切源灰度(cohort 首个成员,建议 pi 或 dsh 之一) | 灰度会话读出站来自新表 | T5/T10/T11/T12 + A/B 钉常开 + tripwire 抽样在场;exit switch 一键回退 | +| **P3** | decompress/export/UI/计费逐一切换;CCR/persona 处置;#3756 退役判断;cohort 名单扩员 | 消费方逐个切换 | T8 + 全矩阵复跑;每期独立 PR | + +**E 裁定建议维持**:本期范围到 P0+P1 收手交验收;P2 的字节对等风险集中,独立 PR + 灰度基建齐备后再上。 + +--- + +## 13. 前置依赖与时序 + +1. **#5855 先合并**(人合):它是事务骨架的工程基底,也是 schema 版本的垫层。它 open/dirty 一天,P0 的 DDL 就没法干净地叠版本。 +2. #5842(身份采集)✓ 已在 master —— 第 0 段直接复用,无需等待。 +3. #5854(撤回)✓ 已 merged —— 确认 v6 符号零残留于 master。 +4. 裁定闭环(§14):F/G/B 三项决定表结构骨架,先行拍板;其余随 P0 PR 一并裁。 + +--- + +## 14. 待 owner 裁定汇总 + +已在线讨过的(A–E,沿用草案 v0.1 编号)+ 缺口分析新增(F–L)+ 遗留(D3–D5): + +| # | 事项 | 倾向 | 阻塞什么 | +|---|------|------|----------| +| A | 行内形态 | ② 只存 post,pre 引用既有 CAS;附加:被存活行引用的 pre 字节活得比该行久 | P0 DDL | +| **B** | 失效行保留策略 | 绑定覆盖块生命周期的压实清理;反对永不清(#3752 红线)与全局 TTL | P0 DDL/GC 设计 | +| C | 摘要落位 | ① 就地占覆盖段首 pos + ckpt meta 显式保留覆盖映射 | P0 DDL | +| D | 会话键空间 | 沿用现有 sid | P0 DDL | +| E | 本期范围 | P0+P1 收手交验收,P2 另开 PR 灰度 | 排期 | +| **F** | 内核状态居所 | ckpt.meta_json 携带 CompressionState 序列化 + 校验和(内核零改动、水合 O(1)) | P0/P1 核心 | +| **G** | L1 身份薄表 | 恢复 baseline §4.1.1 的 (sess_raw, ledger_id) 持久化(独立薄表 + 索引),复用 #5842 采集 | P0 DDL | +| **H** | 写模型 | 单进程假设(WAL 串行化)or 多进程 + 内存视图失效协议,P0 二选一定稿 | P0 | +| **I** | 硬门 vs 降级边界 | cohort 声称宿主故障拒绝 + 零副作用 + breaker;非 cohort 走顺序>hash 同表降级、会话粘滞、换模式=resync | P0/P2 路由 | +| **J** | tripwire | 运行期低比例抽样 + 不匹配即摘回 legacy + 告警 | P2 | +| **K** | 与 #3756 / v5 共存 | 新底座车道上 #3756 让位退役(旗互斥);v5 继续服务非 cohort | P2/P3 | +| **L** | 行内 token 计账列 | tokens_est 入表:判决面增长数据源 + 计费共用 | P0 DDL | +| D3 | 故障响应形态 | HTTP 码 / 错误体 / 客户端重试约定(R3 硬拒绝的对端视角) | P2 | +| D4 | cohort 初始名单 | shim/pi/opencode/codex/cc/gemini 六候选逐个核实载体稳定性 | P0 配置初值 | +| D5 | 重编号语义 | 内容同 id 异:判新增双份 or 对级去重 | P0 语义 | + +--- + +## 15. 风险与诚实边界 + +- **「保元素数的前缀字节改写不可检测」**(继承 baseline 代价条款):代价就是跳过逐条复核的信任模型变化——需人工评审把关,不得自动合并(KDD §7.4 同类门槛)。 +- **多模态尾部进入缓存后该段历史永久降级 legacy 全量轮**(继承 KDD#15 诚实边界②):宁降级不偏离。 +- **marker-strip 计数随种子账本继承**,中部区从不重扫(同上③)。 +- 本提案所有「不变量」的最终裁判是运行期 tripwire + S4 抽验,不是纸面论证——所以 P1 影子期存在意义就是攒这份实证。 + +--- + +## 16. 参考号速查 + +#5643(本线起源+定位纠正)· #5854(整线撤回,merged)· #5855(工程基底 re-land,open)· #5842(身份采集 wiring,merged)· #3756(D-lane 快照渲染,将被 K 裁定处置)· #3752(存储无界增长 p1 红线)· #5403(OOM:lastMessages 驻留)· #2432(客户端压缩死螺旋,纯忽略策略的来源)· #3236(共享投影判据)· #3642(跨线稳定规范 id)· #3617(gate 双向断言)· #1104/#1106/#1107(fingerprint 白名单纪律)· #4491/#4834(msgid 身份层级)· #5539(schema v11 FK 子索引)· #155(诚实输出)· #360(listen(0) 端口纪律)· MESSAGE-IDENTITY.md(身份教义)· #5643 定位纠正评论: diff --git a/docs/proposals/snapshot-compression-foundation/assets/arch-current.png b/docs/proposals/snapshot-compression-foundation/assets/arch-current.png new file mode 100644 index 0000000000..f731b97a19 Binary files /dev/null and b/docs/proposals/snapshot-compression-foundation/assets/arch-current.png differ diff --git a/docs/proposals/snapshot-compression-foundation/assets/arch-current.svg b/docs/proposals/snapshot-compression-foundation/assets/arch-current.svg new file mode 100644 index 0000000000..87cc009c8a --- /dev/null +++ b/docs/proposals/snapshot-compression-foundation/assets/arch-current.svg @@ -0,0 +1,174 @@ + + + + + + + + + 现状架构 · bili 上下文压缩代理(origin/master,存储 schema v11) + 四条车道各自为政 + kstate JSON 平行真源 —— 本提案要替换的格局(#5854 已把 v6 整线撤出 master) + + + + + 宿主/既有管道 + + 内核(不改) + + 统一存储 + + 结构性痛点 + + 默认主路径 + + 实验性灰度车道 + + + + + 宿主车道(今天已有身份采集) + dsh — shim / 增量信封 + pi — entry-walker 采集 + opencode — transform 钩子 + codex — responses wire + 其他宿主 — 裸代理客户端 + 请求头带 session id;插件车道 + 信封里逐元素带宿主 msgid + (#5842 采集器已就位) + + + + 入站适配器 prepare-* + anthropic | openai | responses | google + + + lane-decisions 分轨 + 按请求形状 × 会话粘滞绑定 + + + + legacy 全量重发路径(今天的默认主路径) + JSON.parse 全量 + openaiToCore 全量转换 + lastMessages AST 钉在内存跨轮驻留(#5403 OOM 生产者) + + + 增量 plugin-turn 车道 + anthropic wire divert · x-bili-plugin-turn 回合 + + + v5 watermark 快路径(5% 灰度) + fastpath-scan 结构扫描 · x-bili-pf-seq 戳握手 + + + D-lane 快照渲染 #3756(≈0% 灰度) + 服务端正史渲染出站前缀 · 失败整体回退 legacy + + + + acp-kernel 压缩内核 + (本提案零改动) + 折叠判定 projectUnit(共享投影) + 摘要生成 + 四条 prompts + nudge flat-50K 固定节拍 + ref/msgid 账本(id 永不复用) + CompressionState: + blocks / refs / deadRefs / ckpt + tokenSnapshot / nextBlockId … + 状态由谁水合、存哪 —— 见下 + + + + 出站映射 coreTo* + 四线协议重新序列化 + carrier 分支: + pluginMode:抑制 tool+nudge + proxy:acp_summary + revoice 为 user 消息 + pf-*/信封头永不出代理 + + 模型端点 + sglang / openai·responses + anthropic / google + + + + 统一 SQLite · schema v11(#5539 FK 子索引)—— 行级真源只有一半 + + + sessions + content + messages + blocks / dead_refs + refs / checkpoints + ccr_entries + kv_store + message_media / ext + + + conversations + session_ext + kstate JSON(hydration 字段清单) + + messages.msgId 列存的是 bili/内核规范 id —— 宿主 msgid 没有落点;sess_raw 无任何表承载。 + CompressionState 的存活字段由 src/state-hydration.ts mergeState() 单一定义,经 persist.ts 走 kstate JSON 存取。 + #5855(事务骨架 re-land)仍 open/冲突未解 —— 新四表 DDL 必须叠在它落地之后。 + + + + ⚠ 痛点① 状态三处住:进程内存 / 表 / kstate JSON + 双写分叉是刚撤回那条线的根病;#5403 OOM 来自 lastMessages 整包驻留。 + 重启靠 mergeState 水合拼装,拼错即降级或烧会话。 + + ⚠ 痛点② 身份不落盘:世界匹配永远冷启动 + 宿主 (sess_raw, msgid) 对 ingest 即剥掉,只在 in-flight 信封里活一轮。 + 「这元素见过没」跨进程/跨重启无法回答 ⇒ 每轮重探测、重武装。 + + ⚠ 痛点③ 快路径各自打补丁,无统一快照表 + v5 水位 / 增量车道 / D-lane 各管一段流量;缓存稳定性靠多层补丁堆叠, + 任何一层失步就是缓存投毒或整体回退。缺一个单一存储真源。 + + + + 一句话总结 → 本提案目标:原始字节 → 协议转换 → 改写(压缩) → 快照表应用 → 出站映射 → 出站,六段固定链每轮全走完,单一存储真源。 + 内核(折判定/摘要/prompts/ref 契约)零改动 · 有 session+msgid 的宿主全走此路 · 无 msgid 按 顺序>hash 降级写同一套表。 + 现状图对应物:左起入站 → 四车道(替代为本链第 0–1 段的身份化分轨)→ 内核(第 2 段,不动)→ 出站(第 4–5 段);新增的第 0 段(L1 身份)与第 3 段(快照表)在现状中不存在。 + + + + + + + + + + + + + + + + + + + + + + + + + 状态读写 = kstate JSON + 表 双通道(分叉点) + diff --git a/docs/proposals/snapshot-compression-foundation/assets/arch-migration.png b/docs/proposals/snapshot-compression-foundation/assets/arch-migration.png new file mode 100644 index 0000000000..82914bb01d Binary files /dev/null and b/docs/proposals/snapshot-compression-foundation/assets/arch-migration.png differ diff --git a/docs/proposals/snapshot-compression-foundation/assets/arch-migration.svg b/docs/proposals/snapshot-compression-foundation/assets/arch-migration.svg new file mode 100644 index 0000000000..69d51ef404 --- /dev/null +++ b/docs/proposals/snapshot-compression-foundation/assets/arch-migration.svg @@ -0,0 +1,35 @@ + + + + + + + + + + + 组件迁移总览 · 「现状」的十块组件在新架构下去向何处 + 同一条链路、同一个内核 —— 多数组件原样保留或收窄职责;变化集中在「状态住哪、身份记在哪」两处收敛到单一存储真源 + 保留 / 复用 / 扩展 · 零改动替换 / 收敛角色变更 / 域收窄让位 · 退役(待裁)红 = 痛点 绿 = 解决锚点 + + 今天 · origin/master(storage schema v11,四车道各自为政) + 底座落地后(本提案 · 六段固定链) + 宿主车道 msgid 采集#5842 四载体已就位(shim/pi/opencode/codex…)保留·复用第 0 段 身份入册输入(采集零改)双信道身份原样进链;新下游 l1_world 落盘 (sess_raw↔ledger_id),跨进程/跨重启恒可答(P8/P9)入站适配器 prepare-*{openai,anthropic,responses,google}ToCore 矩阵保留·复用第 1 段 协议转换复用wire 矩阵不动;span 解析只做身份差集 NEW 元素(任意体位,含 pos 0);散布编码字节等价钉 = P2 准入lane-decisions 分轨请求头 session id + 会话粘滞绑定保留·扩展路由矩阵 · cohort 判定成行每组合一条正负成对行(#3617 gate 双向断言);cohort 边界(故障硬拒绝 vs 顺序>hash 降级)全进判定函数legacy 全量重发路径今天的默认主路径(JSON.parse 全包 + 钉存 AST)角色变更兜底安全网 · 整轮回落通道非 cohort 仍是主路径;cohort 会话命中 R1–R4 任一红门 ⇒ 该轮整体走它并重新盖戳,客户端一轮收敛增量 plugin-turn 车道anthropic-wire divert · x-bili-plugin-turn 回合保留·原职原职维持 · 按请求形状分轨anthropic-wire divert 回合照旧;dsh shim 逐回合与快照底座二选一,服务端不打混合标v5 watermark 快路径5% 灰度 · fastpath-scan 结构扫描保留·域收窄域收窄 · 只服务非 cohort管解析成本、不管状态真源;与快照底座同表不同车道、旗位独立,互不双处理D-lane 快照渲染 #3756≈0% 灰度 · 正史渲染出站前缀让位·退役让位 · 旗位互斥(待裁定 K)在新底座车道上与 snap 三件套旗位互斥运行;退役与否由 P3 依影子期数据裁定⚠ 待裁定 Kacp-kernel 压缩内核projectUnit / prompts / nudge flat-50K / ref 账本零改动零改动 · I/O 契约扩展为五事件流cast / append / update / delete / compact;折叠判定、四条 prompts、flat-50K nudge、id 永不复用合同一行不动出站映射 coreTo*双 carrier 分支 · pf-/信封头永不出代理替换mapEgress + tripwire 抽样存活行按 pos 排序 ++ 本轮尾窗 [K..T) 新解析;carrier 分支逻辑照搬(#377 教训);抽样不一致 ⇒ 立即摘回 legacy + 告警状态存储:kstate JSON + 表族v11 · hydration 水合拼装(mergeState 单一事实源)收敛单一存储真源 · 三件套 + L1snap_rows/snap_log/snap_ckpt 单事务提交(不变量 i–v);kstate 降为红道回退源,绿道读三件套 + + 三个结构性痛点 → 一一对应的解决锚点 + 痛点① 状态三处住进程内存 + 表 + kstate 双写分叉;#5403 OOM 钉存来源解决:ckpt.meta_json 单一真源(F 裁定)内存降为缓存;绿道读三件套 / 红道随时承接 kstate 旧路,切换按会话单向、失步整轮回退痛点② 宿主身份不落盘(sess_raw,msgid) ingest 即剥,只在 in-flight 信封活一轮解决:L1 身份薄表持久化(G 裁定)复用 #5842 载体采集器;首见即记、永不截断删除(P8/P9);裁决三分支 echo 抽验 / 复合探测 / 发放新身份痛点③ 快路径各自打补丁v5/增量/D-lane 各管一段流量,缓存稳定性靠补丁叠加解决:单事务提交原语 + 四件套保护三步原子:upsert rows → append log(seq 严格单调) → compact 可选;tripwire / S4 校验和 / 见证要求 / 审计抽验 + \ No newline at end of file diff --git a/docs/proposals/snapshot-compression-foundation/assets/arch-stages.png b/docs/proposals/snapshot-compression-foundation/assets/arch-stages.png new file mode 100644 index 0000000000..bcf9d58d0e Binary files /dev/null and b/docs/proposals/snapshot-compression-foundation/assets/arch-stages.png differ diff --git a/docs/proposals/snapshot-compression-foundation/assets/arch-stages.svg b/docs/proposals/snapshot-compression-foundation/assets/arch-stages.svg new file mode 100644 index 0000000000..027c6d81e0 --- /dev/null +++ b/docs/proposals/snapshot-compression-foundation/assets/arch-stages.svg @@ -0,0 +1,271 @@ + + + + + + + + + 目标架构 · 快照表式压缩底座(六段固定链,单一存储真源) + 每轮全走完:原始字节 → 协议转换 → 改写(压缩) → 快照表应用 → 出站映射 → 出站 | 内核零改动 | 任何疑点该轮回落 legacy 路径(降级,不猜) + + + + 入站 · 宿主车道与分轨(零改动 —— #5842 msgid 采集已就位;分轨走既有 lane-decisions) + dsh shim + pi entry-walker + opencode transform 钩子 + codex responses 车道 + 其他裸代理客户端 + prepare-* 适配器(wire 矩阵,现成) + + + + + + 底座命中的请求 ⇒ 进入下方六段固定链 + cohort:双信道身份(sess_raw+msgid[]) · 非 cohort:顺序 > hash 粘滞降级,同表写入(见第 0 段) + + 增量 plugin-turn 形状回合 + anthropic wire divert · 维持原职(见下方共存带) + + 其余流量(v5 形状会话 / 底座灰度关) + watermark 快路径或 legacy 全量路径 · 现状机制不变 + 分轨判定函数不新增:提案只在 lane-decisions 加一行「底座命中」正负成对断言行(#3617);任何红门命中 ⇒ 该轮回落 legacy 全量并重新盖戳。 + + + + + + 0 + 入站 · 身份入册 + 新增段(现状不存在) + + 原始字节体(全量重发历史) + 双信道身份: + 请求头 sess_raw + + 逐元素 msgid[] 信封 + #5842 载体 resolver 采集复用 + + L1 世界匹配(新薄表) + l1_world: sess_raw ↔ ledger_id + 全局复合索引,原值禁截断(P9) + 裁决三分支: + echo 直中 + 抽验 + 复合探测 + 发放新身份(+一轮武装) + + 非 cohort 宿主: + 顺序 > hash 粘滞降级,同表写入 + (pos 由序位发号;换模式=resync) + P8 id 永生:首见即记、跨重启恒可答 + + + + + 1 + 协议转换 → 通用核 + 现成复用(master 已有) + + wire 矩阵 *ToCore + openai/anthropic/responses/google + + 跨线稳定规范 id(#3642) + + span 解析仅 NEW 元素 + 身份差集散布任意体位 + (客户端压缩后摘要可能在 pos 0) + + ClusterCounter 复现钉 + 散布子集解析必须复现整数组 + 调用的 _N 编号上下文 + ⇒ 字节级回归钉(P2 准入) + + 媒体格 → CAS 引用 + base64 不进行表(message_media) + + 合法入口恰两个: + 全新会话 / 新身份发放轮 + (P5 禁全量重解析) + + + + + 2 + 改写 / 压缩 + 内核零改动 + + acp-kernel(不动一行) + projectUnit 共享投影判定 + 摘要 prompts / nudge flat-50K + ref/msgid 账本 id 永不复用 + + 产出 = 结构化事件 + append / update / delete / compact + 按内核管道证据寻址(block 覆盖/ + ledgerSeq/帧边界),禁模糊相似 + + 时序(附录二) + 第一现场 POST /__bili/plugin/tool + 当次请求内同步单事务落表 + 下一轮模型请求只做核对 + 绕过宿主(dsh native compaction) + ⇒ checkpoint 帧签名+rebase 检测 + 喂同一例程兜底 + + 窗口期挂 structural-pending 红门 + + + + + 3 + 快照表应用 + 新增核心(本提案主体) + + 提交原语 = 单事务 + upsert snap_rows → 追加 snap_log + (seq 严格 > max(log.max_seq, + ckpt_seq)) → 可选折 ckpt + 抛错 = 整轮放弃,不落半态 + + snap_rows + snap_log + snap_ckpt + l1_world(L1) + + 行键 = ledger 绝对 pos + append-only 永不复用 + ckpt.meta_json 三件套 + 内核态序列化(F) + 覆盖映射(C) + + S4 校验和 + + 失效行 lifecycle(B) + inactive 保留双形态供 decompress; + 物理清绑覆盖块生命周期(#3752 红线) + + + + + 4 + 出站映射 + mapEgress(读侧重建) + + 存活行按 pos 排序 + ++ 尾窗 [K..T) 本轮新解析 + 前缀跨轮字节稳定 + ⇒ 前缀缓存友好(目标函数) + + 双 carrier 分支(粘滞) + pluginMode:抑制 tool+nudge 注入 + (agent 本地执行压缩) + proxy 模式:acp_summary + revoice 为 user 消息 + + 不是红条件的变化: + 跨协议族切模型 / maxDim 变 / + toolArmed 变 = 只动映射输入 + (快照只存改写右侧通用核形态) + + 失步处理 + 链断/join 不上 ⇒ 该轮整条走 + legacy 全量路径并重新盖戳 + + + + + 5 + 出站 + tripwire 守卫 + + tripwire 抽样比对 + 出站字节 vs legacy 全量产物 + (除 time attr),低比例运行期采样 + 一致 → 放行上游 + + 不一致(J): + 该会话立即摘回 legacy + 告警 + 防漂移状态供出陈旧前缀=缓存投毒 + + + 模型端点 + sglang / openai·responses + anthropic / google + + 协议卫生 + pf-*/信封握手头永不出到模型端点 + (UPSTREAM_HOP_HEADERS 白名单纪律) + + 灰度期全程 A/B 字节钉在场 + + + + + + + + + + + + + + 降级红门枚举(任一命中 ⇒ 该轮/该会话整体回落 legacy 全量路径并重新盖戳 —— 降级不猜) + + R1 S4 / 校验和坏 + 该 sid 平面永久关死,整体回落旧路 + (底材损坏不可恢复,红线不变) + + R2 出站 join 链断 + core_ids ↔ 行对不上(链回溯失败) + ⇒ 该轮不产事件,整轮 legacy + + R3 cohort 宿主身份门故障(I) + 声称有双信道的宿主某轮收不齐 id + ⇒ 硬拒绝 + 零副作用 + 喂 breaker + + R4 structural-pending 未消费 + 压缩事件到落表之间窗口未闭合;重启丢失 + 标记 = 保守判死(fail-safe 方向) + 注:跨协议族切模型 / maxDim / toolArmed 变化【不是】红条件 —— 它们是每轮映射输入。 + + + + 共存路由(每种流量只有一个家,互不双处理) + v5 watermark 快路径 → 继续服务非 cohort(无 msgid)流量;与本底座同表不同车道,旗位独立。 + 增量 plugin-turn 车道 → anthropic wire divert 回合维持原职(dsh shim 逐回合二选一)。 + 绕过宿主(dsh native compaction 等) → checkpoint 帧签名 + rebase 检测喂第 2 段同一事件例程。 + #3756 D-lane → 新底座就绪的车道让位退役(旗互斥,待裁定 K);lane-decisions 矩阵每组合配正负成对断言。 + + 回滚与灰度 + + 回滚 = DROP 四表 + 关旗 = 逐字节回到今天(无迁移负担,老库升级纯 CREATE IF NOT EXISTS) + rollout:master ⊃ lane 两级嵌套开关(config-file key,knobs env>file>default, + 不新增 env 旋钮);cohort carrier 名单配置级声明;Web「配置」页三态开关同步出厂。 + 默认 OFF,按会话分桶;P2 单车道灰度起 A/B 钉常开。 + + + + 新增段 + 现成复用 + 内核(零改动) + 存储(三表+L1) + 红门/降级 + 回滚保证 + + 基线不变量:P5 禁全量重解析(合法入口恰两个) · P6 持久不重建(重启只水合,无武装轮) · P8 (sess_raw,ledger_id) 首见即记、永不删改 · P9 身份层原值禁截断(sig/fingerprint 只进诊断流) · tripwire/S4/见证/审计抽验四件套保护。 + 目标函数排序:稳定性 / 扩展性 / 通用性优先,性能是副产品 —— 先问流转对不对,不问快不快。 + diff --git a/docs/proposals/snapshot-compression-foundation/assets/arch-target.png b/docs/proposals/snapshot-compression-foundation/assets/arch-target.png new file mode 100644 index 0000000000..68f7bf511a Binary files /dev/null and b/docs/proposals/snapshot-compression-foundation/assets/arch-target.png differ diff --git a/docs/proposals/snapshot-compression-foundation/assets/arch-target.svg b/docs/proposals/snapshot-compression-foundation/assets/arch-target.svg new file mode 100644 index 0000000000..51a0c1d4b3 --- /dev/null +++ b/docs/proposals/snapshot-compression-foundation/assets/arch-target.svg @@ -0,0 +1,189 @@ + + + + + + + + + 目标架构 · 快照表式压缩底座(全貌 —— 与现状图逐位对齐,可直接对照) + 灰 = 零改动 | 青 = 新增段/表 | 琥珀 = 角色变更 | 红虚线框 = 兜底安全网 | 内核零改动 | 任何疑点该轮回落 legacy(降级,不猜) + + + + + 宿主/既有管道 + + 零改动 + + 新增段/表 + + 角色变更 + + 兜底安全网 + + 统一存储 + + + + + 宿主车道(零改动) + dsh — shim / 增量信封 + pi — entry-walker 采集 + opencode — transform 钩子 + codex — responses wire + 其他宿主 — 裸代理客户端 + 请求头带 session id;插件车道 + 信封里逐元素带宿主 msgid + #5842 采集已就位 —— 原样复用 + + + + 入站适配器 prepare-* + anthropic | openai | responses | google(现成复用) + + + lane-decisions 分轨 + 按请求形状 × 会话粘滞绑定 + +一行「底座命中」正负成对断言(#3617) + + + + 快照表底座 · 六段固定链(新主路径) + 身份入册 → 协议转换 → 改写 → 快照表 → 出站映射 + cohort 双信道身份 · 非 cohort 顺序>hash 粘滞降级同表 + + + 增量 plugin-turn 车道(共存 · 维持原职) + anthropic wire divert · dsh shim 逐回合二选一 + + + v5 watermark 快路径(共存 · 旗位独立) + 继续服务非 cohort 流量的解析成本 + + + D-lane 快照渲染 #3756(让位 · 待裁定 K) + 新底座就绪后车道让位退役 · 旗互斥 + + + legacy 全量重发路径(降为兜底安全网) + 任何红门 / 失步命中 ⇒ 该轮整条回到这里并重新盖戳 + + + + acp-kernel 压缩内核 + (本提案零改动 —— 不动一行) + 折叠判定 projectUnit(共享投影) + 摘要生成 + 四条 prompts + nudge flat-50K 固定节拍 + ref/msgid 账本(id 永不复用) + CompressionState(字段集不变): + blocks / refs / deadRefs / ckpt … + 序列化入 snap_ckpt.meta_json 单一水合源 + 状态存哪、谁水合 —— 见下存储带 + + + + 出站映射 mapEgress + 存活行按 pos 排序 ++ 尾窗 [K..T) + 前缀跨轮字节稳定 ⇒ 缓存友好 + carrier 分支粘滞(pluginMode/proxy) + pf-*/信封头永不出代理 + + 模型端点 + sglang / openai·responses + anthropic / google + + + + 统一 SQLite · schema v11 底座 —— 既有 10 表写语义不变;新增四表纯加性(#5855 落地后取新版本) + + + sessions + content + messages + blocks / dead_refs + refs / checkpoints + ccr_entries + kv_store + message_media / ext + + + conversations + session_ext + kstate JSON(仅 legacy 路径保留) + + + + l1_world · sess_raw↔ledger_id + snap_rows · 绝对 pos append-only + snap_log · 事件流 seq + snap_ckpt · meta 三件套(F+C+S4) + + 新增四表 = CREATE IF NOT EXISTS 纯加性、零迁移;回滚 = DROP 四表 + 关旗 = 逐字节回到今天。 + messages/blocks/dead_refs 成为只读审计源(S4 校验和基线);persona fork / conversations / kv_store 不动。 + #5855(事务骨架 re-land)仍 open —— P0 DDL 必须叠在它之后;P1 影子写期行为零变化。 + + + + ✔ 零改动区 —— 内核 / 宿主 / 适配器 / 审计源 + acp-kernel 不动一行;#5842 采集、prepare-*、carrier 规则原样复用; + 既有 10 张统一存储表写语义不变(messages/blocks 只作审计源)。 + + + 新增区 —— 四薄表 + 两个新段 + L1 身份薄表(sess_raw→ledger_id,P8 id 永生) + snap 三件套(rows/log/ckpt); + 新段 = 入站身份入册 + 快照表应用(内部细节见分阶段展开图)。 + + ⇄ 角色变更区 —— 三处降级,其余维持原职 + legacy 全量 → 兜底安全网 · D-lane #3756 → 让位(待裁定 K) · kstate JSON → legacy 专属; + v5 / 增量 plugin-turn 维持原职不变,每种流量只有一个家、互不双处理。 + + + + 一句话影响范围 = lane-decisions +1 行 · 一条新链(六段底座) · 四张新表 · 三处角色变更 —— 其余组件与现状图同形同位。 + 降级红门 R1–R4:S4 坏=平面永久关死 · join 断=该轮不产事件 · cohort 身份门故障=硬拒绝+零副作用 · structural-pending 未消费=保守判死(细节见分阶段展开图)。 + rollout:master ⊃ lane 两级嵌套开关(config-file key,不新增 env 旋钮,默认 OFF 按会话分桶) · 灰度期 A/B 字节钉常开 · Web「配置」页三态开关同步出厂。 + + + + + + + + + + + + + + + + + + + + + + + + + + + 状态读写 = snap_ckpt.meta_json 单一真源(kstate 双通道分叉点消除) + diff --git a/docs/proposals/snapshot-compression-foundation/assets/gen-migration.mjs b/docs/proposals/snapshot-compression-foundation/assets/gen-migration.mjs new file mode 100644 index 0000000000..55d270cab0 --- /dev/null +++ b/docs/proposals/snapshot-compression-foundation/assets/gen-migration.mjs @@ -0,0 +1,88 @@ +import fs from 'node:fs'; +const W = 1560, H = 1050; +const rows = [ + { l:['宿主车道 msgid 采集','#5842 四载体已就位(shim/pi/opencode/codex…)'], f:'保留·复用', c:'g', r:['第 0 段 身份入册输入(采集零改)','双信道身份原样进链;新下游 l1_world 落盘 (sess_raw↔ledger_id),跨进程/跨重启恒可答(P8/P9)'] }, + { l:['入站适配器 prepare-*','{openai,anthropic,responses,google}ToCore 矩阵'], f:'保留·复用', c:'g', r:['第 1 段 协议转换复用','wire 矩阵不动;span 解析只做身份差集 NEW 元素(任意体位,含 pos 0);散布编码字节等价钉 = P2 准入'] }, + { l:['lane-decisions 分轨','请求头 session id + 会话粘滞绑定'], f:'保留·扩展', c:'g', r:['路由矩阵 · cohort 判定成行','每组合一条正负成对行(#3617 gate 双向断言);cohort 边界(故障硬拒绝 vs 顺序>hash 降级)全进判定函数'] }, + { l:['legacy 全量重发路径','今天的默认主路径(JSON.parse 全包 + 钉存 AST)'], f:'角色变更', c:'a', r:['兜底安全网 · 整轮回落通道','非 cohort 仍是主路径;cohort 会话命中 R1–R4 任一红门 ⇒ 该轮整体走它并重新盖戳,客户端一轮收敛'] }, + { l:['增量 plugin-turn 车道','anthropic-wire divert · x-bili-plugin-turn 回合'], f:'保留·原职', c:'g', r:['原职维持 · 按请求形状分轨','anthropic-wire divert 回合照旧;dsh shim 逐回合与快照底座二选一,服务端不打混合标'] }, + { l:['v5 watermark 快路径','5% 灰度 · fastpath-scan 结构扫描'], f:'保留·域收窄', c:'a', r:['域收窄 · 只服务非 cohort','管解析成本、不管状态真源;与快照底座同表不同车道、旗位独立,互不双处理'] }, + { l:['D-lane 快照渲染 #3756','≈0% 灰度 · 正史渲染出站前缀'], f:'让位·退役', c:'gr', tag:'待裁定 K', r:['让位 · 旗位互斥(待裁定 K)','在新底座车道上与 snap 三件套旗位互斥运行;退役与否由 P3 依影子期数据裁定'] }, + { l:['acp-kernel 压缩内核','projectUnit / prompts / nudge flat-50K / ref 账本'], f:'零改动', c:'k', r:['零改动 · I/O 契约扩展为五事件流','cast / append / update / delete / compact;折叠判定、四条 prompts、flat-50K nudge、id 永不复用合同一行不动'] }, + { l:['出站映射 coreTo*','双 carrier 分支 · pf-/信封头永不出代理'], f:'替换', c:'i', r:['mapEgress + tripwire 抽样','存活行按 pos 排序 ++ 本轮尾窗 [K..T) 新解析;carrier 分支逻辑照搬(#377 教训);抽样不一致 ⇒ 立即摘回 legacy + 告警'] }, + { l:['状态存储:kstate JSON + 表族','v11 · hydration 水合拼装(mergeState 单一事实源)'], f:'收敛', c:'i', r:['单一存储真源 · 三件套 + L1','snap_rows/snap_log/snap_ckpt 单事务提交(不变量 i–v);kstate 降为红道回退源,绿道读三件套'] }, +]; +const pains = [ + ['痛点① 状态三处住','进程内存 + 表 + kstate 双写分叉;#5403 OOM 钉存来源','解决:ckpt.meta_json 单一真源(F 裁定)','内存降为缓存;绿道读三件套 / 红道随时承接 kstate 旧路,切换按会话单向、失步整轮回退'], + ['痛点② 宿主身份不落盘','(sess_raw,msgid) ingest 即剥,只在 in-flight 信封活一轮','解决:L1 身份薄表持久化(G 裁定)','复用 #5842 载体采集器;首见即记、永不截断删除(P8/P9);裁决三分支 echo 抽验 / 复合探测 / 发放新身份'], + ['痛点③ 快路径各自打补丁','v5/增量/D-lane 各管一段流量,缓存稳定性靠补丁叠加','解决:单事务提交原语 + 四件套保护','三步原子:upsert rows → append log(seq 严格单调) → compact 可选;tripwire / S4 校验和 / 见证要求 / 审计抽验'], +]; +const col = { g:['#dcfce7','#16a34a'], i:['#e0e7ff','#4f46e5'], a:['#fef3c7','#b45309'], gr:['#f1f5f9','#64748b'], k:['#ffffff','#16a34a'], rd:['#fee2e2','#b91c1c'] }; +const mark = { g:'arrG', i:'arrI', a:'arrA', gr:'arrGr', k:'arrG' }; +const fcol = { g:'#15803d', i:'#4338ca', a:'#b45309', gr:'#475569', k:'#15803d' }; +let s = ''; +const rowH = 52, pitch = 64, y0 = 158; +rows.forEach((row, i) => { + const y = y0 + pitch * i; + const [fill, stroke] = col[row.c]; + const sw = row.c === 'k' ? ' stroke-width="2.5"' : ''; + s += ``; + s += `${row.l[0]}`; + s += `${row.l[1]}`; + s += ``; + s += `${row.f}`; + s += ``; + s += `${row.r[0]}`; + s += `${row.r[1]}`; + if (row.tag) s += `⚠ ${row.tag}`; +}); +let p = ''; +pains.forEach((pn, j) => { + const y = 846 + j * 56; + p += ``; + p += `${pn[0]}`; + p += `${pn[1]}`; + p += ``; + p += ``; + p += `${pn[2]}`; + p += `${pn[3]}`; +}); +const legend = [[30,'g','保留 / 复用 / 扩展 · 零改动'],[250,'i','替换 / 收敛'],[380,'a','角色变更 / 域收窄'],[530,'gr','让位 · 退役(待裁)'],[710,'rd','红 = 痛点 绿 = 解决锚点']] + .map(([x,k,txt])=>`${txt}`).join(''); +const svg = ` + + + + + + + + + + 组件迁移总览 · 「现状」的十块组件在新架构下去向何处 + 同一条链路、同一个内核 —— 多数组件原样保留或收窄职责;变化集中在「状态住哪、身份记在哪」两处收敛到单一存储真源 + ${legend} + + 今天 · origin/master(storage schema v11,四车道各自为政) + 底座落地后(本提案 · 六段固定链) + ${s} + + 三个结构性痛点 → 一一对应的解决锚点 + ${p} +`; +fs.writeFileSync('arch-migration.svg', svg); +console.log('written', svg.length, 'bytes; canvas', W + 'x' + H, '; last content y ≈ 1002');