Skip to content

[PR] backport-carry: storage fixes for #4390 / #4444 / #4506 onto the v0.1.206 patch lane - #4577

Merged
ranxianglei merged 6 commits into
bugfix-v0.1.206from
backport/storage-fixes-4390-4444-4506
Oct 24, 2026
Merged

ranxianglei merged 6 commits into
bugfix-v0.1.206from
backport/storage-fixes-4390-4444-4506

Conversation

@ranxianglei

Copy link
Copy Markdown
Owner

Manual carry (the fixes landed on master without PRs, so the backport-label bot path has no PR to label — per the documented fallback in backport.yml: cherry-pick onto a branch, PR against the lane, label backport-carry).

Carried commits (base: v0.1.206)

commit origin fixes
storage: position-independent row identity + historian tombstones f013414 #4390
fix(#4390): drop tailGaps from parity digests + fix stale v7 test pins 1418989 #4390
fix: surface the Node.js upgrade lever on the unsupported-node storage degrade 07f9288 #4444
fix(#4390): full-path remap adopts fresher bytes when identity matches but bytes diverge f0b3e8a #4390
fix(#4506): release media refs on historian retire + budget the #3250 custom-cap slack 8eb194e #4506 (master-CI-red fix — carried so 0.1.207 doesn't ship with a known-red test lane)
docs: release notes for v0.1.207 — release-notes entry (tier: recommended) + CHANGELOG section

package.json stays 0.1.206 on the lane (one-click drift guard requires tree version == npm latest; the bump happens in the ship commit).

Conflict resolutions

CHANGELOG.md hunks only (each pick's entry re-homed under the 0.1.207 section); no code conflicts.

Verification (this exact tree)

  • storage test family tests/storage-*.test.ts: 188/188 green (incl. the 51 tests across the 7 touched files)
  • tsc --noEmit clean
  • cherry-picks validated pairwise: without f0b3e8a the watermark boundary-tamper test is red; without 8eb194e the WAL-cap/media-refs tests are red (this is why the carry is 5 commits, not 3)

After merge

Ship via Release (one-click) → train=✓, version blank (resolves 0.1.207). Requires #4576 (train-mode lane guard) merged first.

ranxianglei and others added 6 commits October 24, 2026 22:25
)

Four p2 fixes from the five-dimension storage review, one PR:

- identity column: cross-save JOIN key strips id/wireIndex (D4-1);
  content_hash stays the byte-faithful CAS key. v8 migration backfills
  and heals wireIndex-split generations (keep lowest seq, refund CAS
  refs). MySQL twin via information_schema probe + idempotent ALTER.
- authoritative sweep retires absent rows (retired_at tombstone)
  instead of DELETE; rewind-redo reactivates in place at the original
  seq (remap covers completeLog-less saves too); readers filter
  retired rows (D4-2).
- snapshot-render seam image signature hashes the bytes (24-hex
  sha256), not the length — same-size screenshots can no longer chain
  a false seam into silent truncation (D4-3).
- fork lineage: parent_session_id from metadata.derivedFromSessionId
  (D3-2).
- gc full vacuum routes through vacuum() (TRUNCATE checkpoint included,
  files actually shrink, D5-1); loadLegacyView counts tail pointer gaps
  and all three load paths warn instead of silently shrinking (D4-5).

Stakes first: tests/storage-review-p2s.test.ts (7), storage family +
golden/snapshot/fold-reconcile/incremental/web regressions green.
Review fix for CI reds on this branch, part 1 of 2.

- loadLegacyView's new tailGaps counter (D4-5) leaked into the shadow-parity digest surface: the expect leg (live session) structurally lacks the key while the got leg (loadLegacyView output) always carries it, so canonicalize()'s key-preserving normalization made every sampled verify report a fake drift. Drop the key from all digests — same class as the existing storageMode precedent (#2671): presence-not-value, and genuine missing rows still diverge the message arrays themselves, so nothing is masked. describeViewDivergence skips it too, keeping drift diagnostics clean (issue3919 forensics pins keys deepEqual).

- Two stale user_version=7 pins the v8 bump missed: tests/storage-ext.test.ts (v3→v7 title/asserts) and tests/storage-ingest-watermark.test.ts (v4→v7 title/assert) now expect v8, matching the updated pin in tests/storage-unified.test.ts.
…e degrade (#4444)

On a Node.js that has neither the builtin node:sqlite (< 22.5) nor a pinned
better-sqlite3 artifact (e.g. Node 21.x, ABI 120), the engine self-bootstrap is
refused before any download and the unified store degrades to legacy-only. The
degrade reason on the unsupported-node path carried the specific coverage detail
but DROPPED the "upgrade Node.js to >= 22.5" lever that the sidecar-open path
already carries — so a Node 21 user read "no engine supports this Node.js" with
no actionable way out.

- runtime-status degradeReason(): append nodeSqliteUpgradeHint() to the
  unsupported-node reason (keeps the #3894 specific-detail-first ordering).
- engine-bootstrap runBootstrap(): carry the same hint on both skip warns so the
  lane logs match the status surface.
- shadow UnifiedShadow sidecar-open catch: stop advising "delete the file to
  reset the index" for a no-engine failure — deleting cannot conjure an engine;
  that advice only fits a genuine open failure (byte-identical otherwise).
- regression test: force process.versions.node=21.6.1 and assert the acp_status
  STORAGE line carries both the coverage detail and the >= 22.5 upgrade lever.

Refs #4444
…s but bytes diverge

Review fix for CI reds on this branch, part 2 of 2.

The D4-1 JOIN migration made remapPlanToAbsolute reuse a seq whenever (msg_id, identity) matched, and INSERT OR IGNORE then silently kept the first generation's bytes for that seq. Any byte divergence under a matching identity became permanently unhealable by full ingest: (a) head truncation — the PR's own flagship scenario — leaves stored rows carrying the old wireIndex generation, so the parity digest never matches the live view again: every 20th save reports drift, the watermark is cleared, the next full save re-runs the same no-op remap, never converges, and after DRIFT_ESCALATE_STREAK consecutive samples #4323 freezes this process's envelope writes (legacy file goes stale = data-loss vector); (b) an externally corrupted CAS row (watermark boundary-tamper test) was healed pre-migration by the byte-faithful join minting fresh rows, and is now kept corrupted forever.

Fix: when the full path reuses an existing seq via the identity join and the stored content_hash differs from the plan row's hash, UPDATE the row in place to the fresher authoritative bytes (CAS ref juggling put-new -> update-ref -> release-old, FK/refcount safe; media mappings untouched because identity includes the {$media:hash} markers). Seq stability is unchanged; the delta path stays append-only. SQLite + MySQL twins. identity.ts doc updated to match; CHANGELOG notes the behavior change.
@ranxianglei ranxianglei added the backport-carry Auto-created copy of a backport-labeled PR, targeting the current bugfix-v<tag> train branch label Oct 24, 2026
@ranxianglei ranxianglei changed the title backport-carry: storage fixes for #4390 / #4444 / #4506 onto the v0.1.206 patch lane [PR] backport-carry: storage fixes for #4390 / #4444 / #4506 onto the v0.1.206 patch lane Oct 24, 2026
@ranxianglei
ranxianglei merged commit f4f8883 into bugfix-v0.1.206 Oct 24, 2026
34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-carry Auto-created copy of a backport-labeled PR, targeting the current bugfix-v<tag> train branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant