Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ Versions follow the merge of a `*_release-v*` branch; CI publishes to npm on tag
## [Unreleased]

### Fixed
- **shadow 摘要漂移的单实例成因(#4392 / #4986)**:`canonicalize` 曾按 `Object.entries` 插入序参与哈希,而 live 腿(kernel 状态,`byRef` 按发号顺序)与 replay 腿(`loadLegacyView` 按账本 `ord` 重建 `messageRefs`)对同一份等值状态给出不同键序 → 单进程自我报告 `[diverged: state]`,连续 3 次即冻结本进程的信封写入并清掉入库水位。现在任意深度按键排序(数组保序,Map/Set 形态不变),跳过的键(去掉 `storageMode`/`tailGaps`)集中成 `canonical.ts` 导出的 `DIGEST_SKIPPED_KEYS`,供流式摘要路径共用;升级文案不再断言「多个 bili 实例」。storage-corpus manifest 随摘要定义重钉(只有 10 个摘要值变化,文件清单/格式不动);storage-render 金标无需改动——那套驱动在 harness 侧已先 `sortObjectKeys`,排键只改 shadow 腿与 corpus 摘要。`tests/issue4986-digest-drift-ref-order.test.ts`(取自 #4997)钉住 #4986 的 folded-corpus 面。
- 出站不变量判官新增 I3 配对不变量(#4862 出口校验缺口):此前判官只对账入站↔出站连续性(I1/I2),折叠区内完全免检,于是「折叠拆断一对 tool_use/tool_result」的出站体拿到 clean fold verdict 直接放行,直到上游用 400 `tool_use ids were found without tool_result blocks` 说出真相。现在每个 tool-call 必须有同 id 的 tool-result 在其后、每个 tool-result 必须有同 id 的 tool-call 在其前,折叠区不再豁免;归因保持“谁的错”职责(宿主自己发来的断对不计 bili、compress 在途调用豁免、id 重铸不可洗白)。纯检测不拦截,受 diagnostics.invariantJudge 旋钮控制。
- 块区间边界在重存时不再回退:折叠后客户端重发的视图不含已覆盖原文,重入库时该批次无法重新解析区间(-1/-1),旧逻辑会把已解析的真实区间抹掉。现在区间证据一旦解析即持久(入库 `CASE` 保留;sqlite/mysql 双后端),未知块保持诚实缺口(#4608)。
- 优雅停机(SIGTERM → dispose)的 writer 标记现在记录 `stoppedAt`:重启流程(杀旧进程→立即重绑)不再对已确认死亡的前任实例误报 [split-brain]——30s 新鲜度回退从此只覆盖无法证明死亡的崩溃场景(#3923;release-gate lane A 代理重启时观察到误报)。
Expand Down
5 changes: 5 additions & 0 deletions scripts/live-files.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,11 @@ export const LIVE_FILES = [
// servers; every socket is listen(0), spawns are mocked, so the zone band
// is belt-and-braces like #3890's — the entry is for the tripwire
"tests/issue4811-dsh-busy-drop-selfheal.test.ts",
// #4982: integration rows spawn bili children against freePort() picks and
// follow each child's REGISTERED port (ladder-tolerant); BASE is an
// in-memory-only value, so no zone band is needed — the entry is for the
// tripwire
"tests/issue4982-draining-twin-ladder.test.ts",
];

/** Band width per live file: the zone ladder drifts +1 per EADDRINUSE retry;
Expand Down
60 changes: 39 additions & 21 deletions src/storage/canonical.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,13 @@ import { createHash } from "node:crypto";
* corpus manifests and shadow checks agree byte-for-byte.
*
* Determinism rules, in order:
* - Object keys → SORTED at every depth (#4392): insertion order is not
* part of the contract. The live leg (kernel state) and the replay leg
* (unified rows rebuilt field by field) legitimately build the same
* values in different orders — messageRefs.byRef after a ref-ledger
* re-alignment, mergeState's spread-merged nudge/stats — so hashing
* insertion order made a single instance report drift against itself.
* Arrays keep their order (order IS content there).
* - Map/Set → tagged JSON-able envelopes (key order = insertion order; the
* producers below insert in deterministic order).
* - Wall-clock timestamps (> 2020-09-01 in ms) → "__TS__" — savedAt,
Expand All @@ -17,38 +24,49 @@ import { createHash } from "node:crypto";
* by a newer build legitimately rewrites it (probe-proven: it is the ONLY
* field the legacy save→reload round-trip changes on the folded corpus
* case), so pinning it would break parity on every version bump.
* - metadata.storageMode → dropped entirely — same class of last-writer
* provenance (#2671), but its PRESENCE is new (corpus pins predate it),
* so masking the value like __VER__ cannot keep old pins green; the key
* is removed from digests altogether.
* - tailGaps (#4390) → dropped entirely — read-time diagnostic added by
* loadLegacyView (count of tail pointers whose row is missing); the live
* legacy leg of any parity comparison structurally lacks the key, so
* digesting its presence would report drift on every sampled save. Same
* class as storageMode: the PRESENCE, not the value, is the problem. A
* genuinely missing row still diverges the message arrays themselves.
* - storageMode / tailGaps → in DIGEST_SKIPPED_KEYS, dropped at every depth
* before hashing.
*/
export function canonicalize(value: unknown): unknown {
return canonicalizeWithContext(value, undefined);
}

/**
* Keys that are NEVER part of a digest, at any depth — the ONE skip set every
* canonical digest producer must use (#4392). `canonicalize` below and the
* streamed `canonicalDigest` (#4532) hash the same values through different
* code paths; two hand-maintained skip lists drifted apart once already, which
* is what makes a golden-corpus pin and a shadow parity check read two
* different canonical forms of the same session. Import this table instead of
* re-listing the keys.
*
* - `storageMode` — last-writer provenance (#2671): the corpus pins predate
* the stamp and a re-save legitimately (re)writes it; unlike biliVersion its
* PRESENCE is new, so masking the value like __VER__ cannot keep old pins
* green — the key is removed from digests altogether.
* - `tailGaps` — read-time gap-count metadata added by loadLegacyView (#4390);
* the live legacy leg of any parity comparison structurally lacks the key,
* so digesting its presence would report drift on every sampled save. Same
* class as storageMode: the PRESENCE, not the value, is the problem. A
* genuinely missing row still diverges the message arrays themselves.
*/
export const DIGEST_SKIPPED_KEYS: Readonly<Record<string, true>> = { storageMode: true, tailGaps: true };

function canonicalizeWithContext(value: unknown, key: string | undefined): unknown {
if (Array.isArray(value)) return value.map((v) => canonicalizeWithContext(v, undefined));
if (value instanceof Map) return { __map__: [...value.entries()].map(([k, v]) => [k, canonicalizeWithContext(v, undefined)]) };
if (value instanceof Set) return { __set__: [...value.values()].map((v) => canonicalizeWithContext(v, undefined)) };
if (value && typeof value === "object") {
const out: Record<string, unknown> = {};
for (const [k, v] of Object.entries(value)) {
// storageMode is last-writer provenance (#2671): the corpus pins
// predate the stamp, and a re-save legitimately (re)writes it —
// unlike biliVersion its PRESENCE is new, so masking the value
// is not enough; drop the key from digests entirely.
if (k === "storageMode") continue;
// tailGaps (#4390) is read-time gap-count metadata from
// loadLegacyView — the live legacy leg structurally lacks the
// key, so digesting its presence would flag drift on every
// sampled save; drop it like storageMode.
if (k === "tailGaps") continue;
const record = value as Record<string, unknown>;
// Sorted keys per the #4392 rule in this file's header — see there for
// why insertion order cannot be part of the contract.
for (const k of Object.keys(record).sort()) {
const v = record[k];
// One skip table for every digest producer (#4392): see
// DIGEST_SKIPPED_KEYS for why presence, not value, is the problem.
// hasOwn, not `in` — a "__proto__" key must not read as skipped.
if (Object.hasOwn(DIGEST_SKIPPED_KEYS, k)) continue;
out[k] = canonicalizeWithContext(v, k);
}
return out;
Expand Down
18 changes: 10 additions & 8 deletions src/storage/shadow.ts
Original file line number Diff line number Diff line change
Expand Up @@ -401,13 +401,15 @@ export class UnifiedShadow {
this.driftedSessions.add(legacy.id);
// #4323: count CONSECUTIVE sampled drifts — membership alone
// caps at "repeat yes/no" and cannot distinguish 3+ samples.
// N in a row proves structural divergence (multi-writer
// #394): escalate ONCE at the threshold with a loud WARN
// naming both sides' fold counts, then freeze THIS
// process's envelope writes for the session so the ping-
// pong stops churning the file copy (persist.ts consults
// envelopeFrozen on every save route; the unified store
// keeps receiving every save either way).
// N in a row marks a STRUCTURAL divergence: the watermark
// clear healed nothing. Note the drift can be a single-
// instance artifact (#4392 — digest shape), not only the
// multi-writer #394 case; escalate ONCE at the threshold
// with a loud WARN naming both sides' fold counts, then
// freeze THIS process's envelope writes for the session so
// the ping-pong stops churning the file copy (persist.ts
// consults envelopeFrozen on every save route; the unified
// store keeps receiving every save either way).
const streak = (this.driftStreaks.get(legacy.id) ?? 0) + 1;
this.driftStreaks.set(legacy.id, streak);
if (streak === DRIFT_ESCALATE_STREAK) {
Expand All @@ -420,7 +422,7 @@ export class UnifiedShadow {
} catch {
// counts must never break the heal path
}
loggerLog("warn", `[storage] DIGEST DRIFT ESCALATED for ${legacy.id}: ${streak} consecutive sampled drifts — the views have stopped converging${sides}. This usually means MULTIPLE bili instances share one sessions dir (#394 anti-pattern; second-instance detection: #4177). Freezing THIS process's JSON envelope writes for the session until a clean sample re-admits it — the unified store keeps receiving every save (#4323)`);
loggerLog("warn", `[storage] DIGEST DRIFT ESCALATED for ${legacy.id}: ${streak} consecutive sampled drifts — the views have stopped converging${sides}. Both legs are built in THIS process, so a persistent gap is a digest-shape mismatch first (field only one leg builds, ordering) and only then a second writer: a second bili instance sharing one sessions dir (#394 anti-pattern; second-instance detection: #4177) is one cause among others (#4392). Freezing THIS process's JSON envelope writes for the session until a clean sample re-admits it — the unified store keeps receiving every save (#4323)`);
}
// Panels consume {disabled, lastError} — make the sampled
// degradation visible there, not just in the counter/log.
Expand Down
6 changes: 5 additions & 1 deletion tests/codex-concurrent-launch.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,11 @@ function writeScript(root: string, name: string, body: string): string {

async function waitForFile(file: string, timeoutMs: number): Promise<void> {
const deadline = Date.now() + timeoutMs;
while (!fs.existsSync(file)) {
let lastSize = -1;
for (;;) {
const size = fs.existsSync(file) ? fs.statSync(file).size : -1;
if (size > 0 && size === lastSize) return;
lastSize = size;
if (Date.now() > deadline) throw new Error(`timed out waiting for ${file}`);
await new Promise((resolve) => setTimeout(resolve, 50));
}
Expand Down
20 changes: 10 additions & 10 deletions tests/golden/storage-corpus/manifest.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,14 @@
"files": [
"anthropic/127.0.0.1_29e4587af02d1ed9ae4b0054.json"
],
"digest": "8485c8f69693cca987ec3c612bdc31fe429e70fe0fade5e05b9db936f0418311"
"digest": "3026f3d77e59d5076b712faf9599da05f81d84fd997ac6e6631ff906f4f701c3"
},
"tier2": {
"description": "superseded T1 + active T2, distinct `one` views, byte-identical pair pins #401 normalize-on-load",
"files": [
"chat/127.0.0.1_8453ec7a56646bc014ac0d88.json"
],
"digest": "0cfdba77bc1a052c0c3b8170874c4d963273b6580c822c6c9a5ee8f0f3d09b43",
"digest": "01c4e936c7ab078ae8908acf048ccfccf2ce6896894cc6f33a6174d6ce62fe57",
"notes": {
"t2Block": "b3"
}
Expand All @@ -21,7 +21,7 @@
"files": [
"anthropic/127.0.0.1_e17d4db40407d10867daeb1d.json"
],
"digest": "de057f87e0737222362d44326ec9e139457d50d67566ffaf5a7bb590b549c12d",
"digest": "44aecbb906a425dce328f27b1966af773dcaa7033b2b724af076ed5e4e3bee35",
"notes": {
"ghostRef": "m00010"
}
Expand All @@ -31,14 +31,14 @@
"files": [
"chat/127.0.0.1_bde31987d03ca1ebaeb96ba0.json"
],
"digest": "edf7ac9a0cb912acc39b17667dfd1245a24b6bf6f01b57f510f782258e3e3bcb"
"digest": "5737b5edc484bdd27fedb921a3ffc2b9f6f8334104287e73dcba5f8c3bb57b20"
},
"fork-retained": {
"description": "publicSnapshotRetained only — pluginSnapshot persisted, forkContentStore DROPPED (pins the #2675 ② asymmetry verbatim)",
"files": [
"chat/127.0.0.1_dec59ddccc0b4b89ff346c9b.json"
],
"digest": "63328d62ba50899927adda2154185ec83b9071ca8a84611dec1c57d5d4a51a8e",
"digest": "d7894c369566ec22d6e6a581fa7de9e5f9ad88b09aa6617e0385318c4dde8e6d",
"notes": {
"expectContentStoreOnLoad": "absent"
}
Expand All @@ -48,27 +48,27 @@
"files": [
"anthropic/127.0.0.1_b139318d5755710735cbf443.json"
],
"digest": "deec17f777d11294316b2250160b0577756b2e686ed0f2f1a6441ed06a581dc2"
"digest": "a65628e52318b5044e5007e1e8055c4a273b0ff4e6be8ed18dca4525db491ced"
},
"ccr": {
"description": "content-store.json sibling with two CCR originals (#1097 envelope)",
"files": [
"responses/127.0.0.1_0787df32f3cf98f922f7c0f0.content-store.json",
"responses/127.0.0.1_0787df32f3cf98f922f7c0f0.json"
],
"digest": "2d15c631cf8ee6fd406fb4354db508bed86f11ecaefcb7c9b07c48ab3a7493d8"
"digest": "20d534a2774be30e4f19dccdcb66d6af80b4c7ba2c06f855763954bf8d0c9f72"
},
"twinfile": {
"description": "namespaced v2 record + stale FLAT v1 twin for the same id (#2671 ① twin-namespace)",
"files": [
"6fe5d8f2afee3e83e5360fa8.json",
"openai/127.0.0.1_6fe5d8f2afee3e83e5360fa8.json"
],
"digest": "fe94b9453f3e7e2b57831965fbf87ff2604efdd11e74ecec612cbefd82e0e122",
"digest": "0cab4c22cf71ccd9d91402c3de1006bc2b40bd56aee1abead63f26c798187535",
"notes": {
"flatTwinRequests": 1,
"namespacedRequests": 42,
"flatFallbackDigest": "73ae30ed8e603bd51978f2d891acb6aada8c4769e05acbbfdd2a64d4e1f89e57",
"flatFallbackDigest": "64820a36fdf7ecf4b2f0e053dae33ac5c53df867f067657340a4cec064d804d8",
"flatFallbackRequests": 1
}
},
Expand All @@ -77,6 +77,6 @@
"files": [
"anthropic/127.0.0.1_29e4587af02d1ed9ae4b0054.json"
],
"digest": "8485c8f69693cca987ec3c612bdc31fe429e70fe0fade5e05b9db936f0418311"
"digest": "3026f3d77e59d5076b712faf9599da05f81d84fd997ac6e6631ff906f4f701c3"
}
}
Loading
Loading