Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,21 @@ Versions follow the merge of a `*_release-v*` branch; CI publishes to npm on tag

## [Unreleased]

## [0.1.207] — 2026-10-24

### Storage — 行身份与史官化(#4390,storage review 四个 p2 + 三个顺路 p3)

- **行身份去污染(D4-1)**:统一存储 messages 行新增 `identity` 列——跨保存 JOIN 键改为**位置无关**序列化(剥离 `id`/`wireIndex` 后哈希)。此前身份掺入 wire 位置元数据,客户端头部截断一条消息就会让后续全部行换血(旧行被清扫、seq 重铸、#4335 的区间账记随之抖动)。`content_hash` 保持字节忠实 CAS 键不变。存量库 v8 迁移:回填 identity 并合并 wireIndex 分裂世代(保最低 seq,重复行退还 CAS 引用)。MySQL 同款(information_schema 探测幂等 ALTER + 回填)。
- **权威集清扫改墓碑(D4-2)**:proxy 会话每轮全量保存不再物理 DELETE 缺席行,改为 `retired_at` 墓碑——rewind/undo 的历史在统一存储里**耐久保留**;同一身份回来时原地复活(保原 seq,含 completeLog-less 保存路径 remap 复活)。读面(getMessages/统一重放/视图)默认过滤 retired;CAS 引用不释放。deleteSession/archive/GC 继续拥有物理生命周期。
- **渲染接缝图片真指纹(D4-3)**:snapshot-render 的 identitySig 图片段从 `长度` 改为 `sha256(base64) 前 24 hex`——同尺寸截图不再可能连击假接缝造成静默截断。
- **fork 谱系入正史(D3-2)**:sessions 行的 `parent_session_id` 现在从信封 `metadata.derivedFromSessionId` 落库(此前恒 null,谱系只活在信封里,信封丢失=家族失忆)。
- **顺路**:gc full vacuum 走 `vacuum('full')`(带 TRUNCATE checkpoint,文件真正缩小,D5-1);loadLegacyView 尾指针缺行从静默缩水改为计数 + 三处加载路径 warn(D4-5);fold-reconcile 的 Pass 0 位置副本失步(存储 ids 与顺序骨架不再 1:1)从静默降级改为响亮 warn(此前无任何日志痕迹)。
- **审查修复(review)**:identity JOIN 迁移后,全量 ingest 对「identity 相同、字节不同」的行原本靠 INSERT OR IGNORE 静默保留旧世代——头部截断后存储世代带着旧 wireIndex 与活视图永久失配(采样校验漂移→水位反复清零→#4323 信封冻结的数据丢失向量),外部 CAS 损坏也永远无法自愈。现在全量路径在**同一 seq** 上采纳更新的字节(seq 稳定性不变,delta 追加路径仍只增不改);`tailGaps` 诊断键从 parity 摘要面剔除(存在性差异不再误报漂移)。
- 测试:`tests/storage-review-p2s.test.ts` 7 桩(截断保号/墓碑+复活/接缝指纹/谱系/迁移去重/gc checkpoint/缺行响亮);storage 全家族 + golden/snapshot/fold-reconcile/incremental/web 回归绿。

### Fixed

- **master CI 全红修复:#4390 historian 的媒体引用释放 + #3250 WAL cap 断言预算(#4506)。** bisect 定位 f161c518..02f544da 窗口 first bad = #4390(row-identity-historian),两个破口:① 权威集清扫改墓碑后,retired 行的 message_media 映射与媒体 CAS 引用永远不释放(默认要等 archive 365 天)——图片重度会话里即 #3762/#3901 的字节泄漏,`media-cas` 家族在 windows 分片稳定红。现在 retire 时释放媒体映射+引用(文本字节照旧史官保留;图片体积大、历史价值低),rewind-redo 复活时由重摄取路径幂等重建映射与引用(sqlite+mysql 双后端同款)。② `#3250: configured cap` 断言的 CAP_SLACK=4KiB(一页余量)预算不住 #4390 新增的每行 WAL 体积(identity 列 64hex + idx_messages_identity 索引项,~192 行超 12 页/47,376B)——截断不变量本身完好,坏的是页级余量的算术;custom-cap 用例改用 64KiB 的 inter-reset 累积预算(注释引用 #4390 行体积),close-converges 等其余断言维持一页余量不变。
- **增量车道:拉取回执修复 ack 传输盲区 + 漂移回退取富侧(#4352/#4323)**。2026-10-22 事故双根因修复:① pi 的 after_provider_response 只在部分 provider 传输下暴露响应头,x-bili-applied-seq 丢失时 noteIncrementalAck 永不执行,后续每轮都整包 re-seed(sinceSeq=0 重发 21K 条历史)。现在 GET /__bili/plugin/turn?conversation=<id> 返回热 canonical 的 {appliedSeq, boundary}(boundary=末条 wire 消息哈希,冷重建诚实返回 null),pi 扩展在规划下一轮前拉取回执,仅在 boundary 证明 canonical 吸收了我方末条消息时采纳——绝不按 seq 前进盲目采纳(外来的全量重发也会推进 canonical);② #4323 多实例写同一目录时,漂移回退无条件吃旧信封,把索引里更新的折叠世代(26,971 refs/16 active)换成旧世代(11,849/12),覆盖与 canonical 失配→0 可压区间→preflight 8.29M 假性死局 502。现在漂移两侧按 mapped ref 数取富侧(平局信封赢,平局语义=原子整文件)。新增 tests/incremental-ack-pull.test.ts 3 例(响应头 ack 回归钉/拉取采纳/冷 boundary null 拒采纳)+ storage-full-flip 4b(陈旧信封让位新索引世代)。

### Changes
Expand Down
8 changes: 7 additions & 1 deletion release-notes/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,14 @@
},
"billionContextReleaseNotes": {
"schema": 1,
"updated": "2026-10-21",
"updated": "2026-10-24",
"releases": [
{
"version": "0.1.207",
"date": "2026-10-24",
"tier": "recommended",
"summary": "存储可靠性修复(0.1.203–0.1.204 用户请直接升到本版):折叠上下文不再因模型/窗口切换悄悄膨胀直至会话超窗;修复图片会话媒体引用泄漏与存储校验误报(#4390/#4506/#4444)。EN: storage reliability fixes — folded context no longer silently regrows after model/window switches; media-ref leak and false digest drift fixed. v0.1.203–204 users upgrade immediately."
},
{
"version": "0.1.203",
"date": "2026-10-21",
Expand Down
13 changes: 12 additions & 1 deletion src/server/snapshot-render.ts
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,8 @@ const MAX_CHAIN_WALK = 8192;
* MIN_RESYNC_RUN in fold-reconcile (there 3, because reconcile matches on
* position alone; here the second consecutive match must ALSO line up with
* the view, which a stray duplicate cannot provide). */
import { sha256Hex } from "../storage/store.js";

const MIN_SEAM_RUN = 2;

/** #3658 canonical identity signature minus the hash: the tuple the kernel
Expand All @@ -69,10 +71,19 @@ const MIN_SEAM_RUN = 2;
* same bytes ⇒ same length; different images practically always differ). */
function identitySig(m: BiliMessage): string {
const text = typeof m.text === "string" ? m.text : "";
const img = m.imagePart === true ? `|img:${(m.imageBase64 ?? "").length}` : "";
// #4390: hash the image BYTES, never the length — same-size screenshots
// used to collide and chain a false seam (silent truncation). The walk
// only touches the tail region, so per-request hashing is bounded; a
// 24-hex prefix is far beyond collision reach for seam evidence.
const img = m.imagePart === true ? `|img:${sha256Hex(m.imageBase64 ?? "").slice(0, 24)}` : "";
return `${m.role}|${m.contentType}|${m.toolName ?? ""}|${text}${img}`;
}

/** Test seam (#4390): the seam fingerprint under test — see identitySig. */
export function identitySigForTest(m: BiliMessage): string {
return identitySig(m);
}

interface ViewRow {
seq: number;
msg: BiliMessage;
Expand Down
15 changes: 11 additions & 4 deletions src/storage/canonical.ts
Original file line number Diff line number Diff line change
Expand Up @@ -21,10 +21,12 @@ import { createHash } from "node:crypto";
* provenance (#2671), but its PRESENCE is new (corpus pins predate it),
* so masking the value like __VER__ cannot keep old pins green; the key
* is removed from digests altogether.
*
* NOT normalized (deliberately contract): every other byte — ref numbers,
* block ids, summaries, stats, metadata. A change to any of those in the
* round-trip is a migration regression and must fail the golden.
* - tailGaps (#4390) → dropped entirely — read-time diagnostic added by
* loadLegacyView (count of tail pointers whose row is missing); the live
* legacy leg of any parity comparison structurally lacks the key, so
* digesting its presence would report drift on every sampled save. Same
* class as storageMode: the PRESENCE, not the value, is the problem. A
* genuinely missing row still diverges the message arrays themselves.
*/
export function canonicalize(value: unknown): unknown {
return canonicalizeWithContext(value, undefined);
Expand All @@ -42,6 +44,11 @@ function canonicalizeWithContext(value: unknown, key: string | undefined): unkno
// unlike biliVersion its PRESENCE is new, so masking the value
// is not enough; drop the key from digests entirely.
if (k === "storageMode") continue;
// tailGaps (#4390) is read-time gap-count metadata from
// loadLegacyView — the live legacy leg structurally lacks the
// key, so digesting its presence would flag drift on every
// sampled save; drop it like storageMode.
if (k === "tailGaps") continue;
out[k] = canonicalizeWithContext(v, k);
}
return out;
Expand Down
4 changes: 2 additions & 2 deletions src/storage/engine-bootstrap.ts
Original file line number Diff line number Diff line change
Expand Up @@ -353,14 +353,14 @@ async function runBootstrap(opts: EnsureSqliteEngineOptions): Promise<BootstrapO
const spec = specOverride ?? pinnedSpecForThisNode();
if (!spec) {
const detail = unsupportedNodeDetail();
loggerLog("warn", `[storage] sqlite engine self-bootstrap: skipped — ${detail}; staying on legacy storage`);
loggerLog("warn", `[storage] sqlite engine self-bootstrap: skipped — ${detail}; staying on legacy storage${nodeSqliteUpgradeHint()}`);
return { status: "unsupported-node", detail };
}
const unusable = specUnusableReason(spec);
if (unusable) {
// Never register a binding outside the range it declares: the load
// faults uncatchably, so a probe cannot save us — only refusing can.
loggerLog("warn", `[storage] sqlite engine self-bootstrap: skipped — ${unusable}; a self-downloaded engine would crash this process on load, staying on legacy storage`);
loggerLog("warn", `[storage] sqlite engine self-bootstrap: skipped — ${unusable}; a self-downloaded engine would crash this process on load, staying on legacy storage${nodeSqliteUpgradeHint()}`);
return { status: "unsupported-node", detail: unusable };
}
const target = platformOverride ?? platformTargetKey();
Expand Down
102 changes: 102 additions & 0 deletions src/storage/identity.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
/**
* #4390: identity helpers for the unified store.
*
* The messages table carries TWO hashes with different jobs:
* - content_hash: the CAS key of the stored bytes (byte-faithful, includes
* sidecars; putContent refcounts by it). NEVER migrates.
* - identity: the cross-save JOIN key ("is this the message I already
* know?"). Its input STRIPS position metadata — m.id itself and the
* wireIndex the wire codecs stamp on every parse — because those change
* when the client truncates/edits ANYWHERE in the resent history while
* the message's logical content does not (#4390 D4-1: one head-truncation
* used to re-mint the whole surviving generation: new hashes, new seqs,
* block interval bounds built on sand).
*
* Same identity ⇒ same logical message ⇒ same row, same seq. A re-arrival
* with different bytes but the same identity (sidecar drift, a stale
* wireIndex after a head truncation, external CAS corruption) is healed by
* the FULL ingest path, which adopts the fresher bytes at the SAME seq —
* the parity oracle must observe the authoritative view again (#4390
* review); the append-only delta path never rewrites stored rows. A real
* edit changes the content-derived id and therefore the identity.
*/
import { createHash } from "node:crypto";

function sha256Hex(bytes: string | Buffer): string {
return createHash("sha256").update(bytes).digest("hex");
}

/**
* Media payloads below this size ride inline even inside raw sidecars —
* splitting them costs a CAS row + a join for a few hundred bytes (#4069).
*/
const MIN_MEDIA_URL_SPLIT_BYTES = 2048;
/** data:<mime>;base64,<payload> with the payload swapped for its CAS hash. */
const MEDIA_URL_MARKER_RE = new RegExp(
`data:([a-zA-Z0-9.+\\/-]+);base64,([A-Za-z0-9+/=]{${MIN_MEDIA_URL_SPLIT_BYTES},})`,
"g",
);

export function splitMediaContent(content: string): { content: string; hash: string; media: { hash: string; bytes: string }[] } {
let media: { hash: string; bytes: string }[] = [];
let split = content;
if (content.includes("\"imageBase64\"")) {
let parsed: { imageBase64?: unknown };
try {
parsed = JSON.parse(content) as { imageBase64?: unknown };
} catch {
parsed = undefined as unknown as { imageBase64?: unknown };
}
if (parsed) {
const bytes = parsed.imageBase64;
if (typeof bytes === "string" && bytes.length > 0) {
const mediaHash = sha256Hex(bytes);
parsed.imageBase64 = { $media: mediaHash };
split = JSON.stringify(parsed);
media.push({ hash: mediaHash, bytes });
}
}
}
// #4069: raw wire sidecars (rawOpenaiContent / rawOpenaiContentParts /
// rawResponsesItem / rawGoogleParts, …) carry the ORIGINAL part verbatim —
// for image parts that means the same base64 payload a second time, as a
// data URL. Swap every sufficiently large payload for its CAS hash, mime
// prefix and JSON structure untouched, so hydration is byte-identical.
if (split.includes(";base64,")) {
const byHash = new Map(media.map((m) => [m.hash, m.bytes]));
split = split.replace(MEDIA_URL_MARKER_RE, (_all: string, mime: string, payload: string) => {
const mediaHash = sha256Hex(payload);
if (!byHash.has(mediaHash)) {
byHash.set(mediaHash, payload);
media = [...media, { hash: mediaHash, bytes: payload }];
}
return `data:${mime};base64,{$media:${mediaHash}}`;
});
}
return { content: split, hash: sha256Hex(split), media };
}

/** Strip the position/identity metadata from a serialized CoreMessage.
* JSON.parse preserves key insertion order, so the stripped serialization
* is deterministic for the same object shape. */
function identityInputOf(raw: string): string {
let parsed: { id?: unknown; wireIndex?: unknown };
try {
parsed = JSON.parse(raw) as { id?: unknown; wireIndex?: unknown };
} catch {
return raw;
}
if (parsed.id === undefined && parsed.wireIndex === undefined) return raw;
delete parsed.id;
delete parsed.wireIndex;
return JSON.stringify(parsed);
}

/** Identity from a STORED content column (already media-split; may still
* carry id/wireIndex from pre-#4390 bytes). Strip and media-split
* commute, so this equals the identity the live intern pass computes for
* the same message — the migration backfill and the direct-caller
* fallbacks in store.ts rely on that equivalence. */
export function identityOfStoredContent(content: string): string {
return sha256Hex(identityInputOf(content));
}
Loading
Loading