Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,8 +79,10 @@ Explicit `pt-snap import snapshot.pkl --format msinsight --json` produces
`snapshot.pkl.msinsight/` for the fixed msinsight revision's **same original
pickle + adjacent cache** entrance. Inline stacks cost more space than native
interning. Existing compatible targets are never replaced, even with `--force`;
only fully validated identical pt-snap caches are reused. GUI acceptance remains
**pending/not run**; hash equality alone is not acceptance. See
only fully validated pt-snap caches are reused. New exports attest base content so
known device-local Insight cache tables can change without losing reuse; changed
legacy caches without that proof still fail closed. Hash equality alone is not GUI
acceptance; dated fixed-version results and their limits are recorded below. See
[compatible export and limits](docs/en/sharded-snapshotdb.md#explicit-msinsight-compatible-export-p2).

See [offline interoperability acceptance and bounded performance](docs/en/interop-acceptance.md)
Expand Down
4 changes: 3 additions & 1 deletion README_zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,9 @@ pt-snap split snapshot.pkl --slices 4 --output snapshot-slices
显式 `pt-snap import snapshot.pkl --format msinsight --json` 生成
`snapshot.pkl.msinsight/`,对齐固定 msinsight 版本的**同一原始 pickle + 邻接缓存**入口。
内联栈比原生去重占更多空间;即使 `--force` 也不替换已有兼容目标,只复用完整校验且
身份相同的 pt-snap 缓存。GUI **待验收/未运行**,hash 相等不代表验收通过。详见
身份相同的 pt-snap 缓存。新产物的基础内容证明允许已知的本设备 Insight 派生缓存表变化;
缺少该证明的旧产物字节变化仍保守拒绝。hash 相等不代表 GUI 验收通过;
带日期的固定版本实测及限制见下方记录。详见
[兼容导出及限制](docs/zh/sharded-snapshotdb.md#显式-msinsight-兼容导出p2)。

[离线双向验收与有界性能](docs/zh/interop-acceptance.md) 提供固定版本两条链路、
Expand Down
65 changes: 54 additions & 11 deletions docs/en/sharded-snapshotdb.md
Original file line number Diff line number Diff line change
Expand Up @@ -510,17 +510,60 @@ additional ID-validation memory, not snapshot loading, replay, or total import R
`manifestContractVersion=1`, `exportContractVersion=1`, `referenceContractVersion=1`,
`metadataSchemaVersion=1`, `importFormatVersion=1`, and full `targetRevision`.
The `ptSnap` object also contains `metadata`, `omittedDevices`, and `members`
(canonical relative path → finalized SHA256). Whole validated ownership/identity
requires every member/hash and identical per-slice metadata; unknown/external
caches and unrelated files are not adopted. CLI version alone does not invalidate.

Compatible publication is **no-replace even with force**. Identical recognized
pt-snap caches reuse without loading pickle (force does not bypass this rule).
Changed source path/content, options/contracts, corrupted members or GUI-modified
DBs require a new `--output-dir`; no repair, overwrite or deletion of the old
artifact occurs. Exclusive publication and focus compensation use the existing
checked transaction; source SHA256 is rechecked immediately before publish/reuse.
Native force replacement and recovery semantics are untouched.
(canonical relative path → finalized SHA256). New exports additionally store
`memberContent = {"version": 1, "members": {relative_path: proof}}`. Each proof
records every original SQLite schema object's type/name/table/SQL (not physical
rootpage) and every original table's row count and SHA256. This includes dictionary,
metadata, lifecycle references and internal tables. Rows use type-tagged,
length-framed NULL/integer/IEEE-754 real/UTF-8 text/blob encoding, sorted as BLOBs;
duplicates remain significant. Declared IDs are included; incidental hidden rowid
and physical row order are not. SQLite performs the sort; Python streams rows,
but sorting can require temporary storage. This is not a constant-work import.

Byte-identical members retain the fast path without logical scans. On byte drift,
a prior version-1 proof must match every base schema object and row. Only a newly
added ordinary table exactly named `memory_allocation_cache_v1_<member deviceId>`
is permitted; other tables, versions, devices, indexes, views and triggers are
rejected. An originally attested cache-named table is still protected. Stable
header/layout-only changes can also pass when the entire base proof matches.
Unknown/malformed proofs fail closed, including on the byte fast path. Old exports
without a proof remain reusable only with exact member hashes: changed bytes mean
lost integrity evidence, not a diagnosis of corrupted base data. No proof is
backfilled from already modified content. Attestations are integrity baselines,
not authentication of untrusted manifests or origins.

Owned-proof reuse also has a bounded, process-local 16-root LRU. It stores only
completed ownership results, keyed by the canonical root plus complete manifest
and member-byte fingerprint. Warm hits still read/hash every member and manifest
before and after the operation, recheck transport, inventory and full generation,
and return defensive copies; failures evict the entry. `immutable` resolver mode
never makes this ownership cache stat-only. Nothing is written to the dataset,
manifest, sidecars or a persistent receipt. Overview passes its validated dataset
to metadata/ownership checks; the base validation can be reused only after binding
it to the current full content fingerprint. A resolver result alone never proves
ownership or supplies a missing logical proof. Hard-linked files are rejected.

**Cold-process cost remains:** Insight byte changes do not update the original
manifest hash. A new CLI process must therefore scan/sort the base content again
for import reuse and recognized dataset `metadata`/`overview`; the ordinary query
resolver does not perform this ownership proof. Repeated API calls in one process
can avoid logical rescans while retaining byte verification, but that benefit must
not be presented as a speedup for separate CLI invocations. Deep stacks, table size
and SQLite temporary sorting storage affect cold cost; no large-scale latency
bound is promised. Source proof generation likewise scans every original table.

Compatible publication is **no-replace even with force**. Recognized caches with
matching source/options/identity and byte or base-content proof reuse without
loading pickle; force cannot bypass these checks. Source/options changes, base
changes, unrecognized objects and unproved legacy drift require a new
`--output-dir`. No repair, overwrite, deletion or manifest rewrite occurs.
Per-member metadata still must agree with manifest identity. Manifest/member bytes,
full file generation, directory identities, exact inventory and alias/sidecar checks
bracket inspection; writers must already be stopped and members finalized. This
is change detection, not a filesystem lock or live-writer support. Ordinary
read-only compatibility queries retain their separate validation contract.
Exclusive publication/focus compensation and the final source SHA256 recheck are
unchanged, as are native force replacement and recovery semantics.

Inline text costs `sum(len(callstack.encode("utf-8")) for each event)` rather than
one copy per distinct stack. Existing native interning remains available. Conversion
Expand Down
42 changes: 35 additions & 7 deletions docs/zh/sharded-snapshotdb.md
Original file line number Diff line number Diff line change
Expand Up @@ -407,13 +407,41 @@ raw-frame 丢失与字符串 OOM action 强制转换属于已知局限,不应
`outputFormat=compatibility-v1`、`manifestContractVersion=1`、`exportContractVersion=1`、
`referenceContractVersion=1`、`metadataSchemaVersion=1`、`importFormatVersion=1`、完整
`targetRevision`。`ptSnap` 另含 `metadata`、`omittedDevices` 和 `members`(规范相对路径
→ 最终化 SHA256)。整体已校验所有权/身份要求每个成员/哈希及逐片 metadata 相同;
不接管未知/外部缓存或无关文件。CLI 版本变化本身不使缓存失效。

兼容发布 **force 也 no-replace**。已识别、身份相同的 pt-snap 缓存不加载 pickle 即复用,
force 不绕过该规则。源路径/内容、选项/合同变化、成员损坏或 GUI 修改 DB 后须换新
`--output-dir`,不修复、不覆盖、不删除旧产物。排他发布与 focus 补偿使用已有受检事务,
发布/复用前再次核对源 SHA256;原生 force 替换与恢复语义不变。
→ 最终化 SHA256)。新导出还保存 `memberContent = {"version": 1, "members": {相对路径: proof}}`。
proof 记录全部原始 SQLite schema 对象的 type/name/table/SQL(不含物理 rootpage),
以及每张原表的行数和 SHA256,包括 dictionary、metadata、生命周期引用和内部表。
行按带类型标签和长度分隔的 NULL/整数/IEEE-754 浮点/UTF-8 文本/blob 编码,再按 BLOB
排序;重复行仍计入。声明的 ID 被保护,隐式 rowid 和物理行顺序不作为语义。SQLite
负责排序,Python 流式消费;排序可能使用临时存储,不是常数工作量导入。

成员字节相同时保留快速路径,不扫描逻辑内容。字节改变时,必须有事前 version-1 proof
且全部基础 schema/行匹配;只允许新增普通表 `memory_allocation_cache_v1_<该片 deviceId>`,
其他表、版本、设备名、索引、视图和 trigger 均拒绝。原本已纳入证明的同名缓存表仍受保护。
基础证明全量相同时,稳定的文件头/物理布局变化也可通过。未知或非法 proof 即使字节未变
也拒绝。旧产物无 proof 时仍须整文件 hash 相同;字节变化表示完整性证据失效,不等于基础
数据损坏,不从已修改现状补造历史证明。证明是完整性基线,不认证不可信 manifest 或来源。

所有权证明另有进程内最多 16 个 root 的有界 LRU,只缓存完整成功结果;key 绑定规范 root、
完整 manifest 和全部成员字节 fingerprint。warm 命中仍在操作前后读取并 hash 全部文件,
复查 transport、精确清单和完整 generation,返回深拷贝;失败驱逐缓存。resolver 的
`immutable` 模式也不把此所有权缓存降级成仅 stat 信任。不写数据集、manifest、sidecar 或
持久凭据。overview 把已校验数据集传给 metadata/ownership;重新绑定当前完整内容
fingerprint 后才复用基础校验。普通 resolver 结果不代表所有权,也不能补缺失的逻辑证明。
有硬链接别名的文件拒绝复用。

**冷进程成本仍存在:** Insight 修改字节后,原 manifest hash 不会刷新;每个新的 CLI
进程在 import 复用以及可信数据集 `metadata`/`overview` 时仍须重新扫描/排序基础内容。
普通查询 resolver 不做这项所有权证明。同进程重复 API 调用可省逻辑扫描,但仍全量校验
文件字节;不能把 warm API 收益声称为多个独立 CLI 调用的提速。深栈、表大小、SQLite
排序临时存储都会影响冷成本,不承诺大型数据延迟上界。新导出的 proof 生成也扫描全部原表。

兼容发布 **force 也 no-replace**。已识别、源/选项/身份相同、字节或基础内容证明匹配的
pt-snap 缓存不加载 pickle 即复用;force 不绕过检查。源/选项变化、基础内容变化、未知对象、
旧产物未被证明的字节变化须换新 `--output-dir`。不修复、不覆盖、不删除、不改写 manifest。
逐片 metadata 仍须与 manifest 身份相符。检查前后核对 manifest/成员字节、完整文件 generation、
目录身份、精确成员清单及 alias/sidecar;writer 必须已停止、成员已最终化。这是变更检测,
不是文件系统锁或 live-writer 支持。普通兼容数据集只读查询保留独立校验合同。
排他发布、focus 补偿、发布/复用前源 SHA256 复查及原生 force 替换/恢复语义不变。

内联文本成本按每个事件的 UTF-8 调用栈字节求和,不按不同栈各存一份;原生去重仍可用。
转换暂时并存两种布局,可能留下 SQLite free pages;容量不是 RSS/registry/block/磁盘
Expand Down
24 changes: 19 additions & 5 deletions src/pt_snap_cli/core/import_metadata.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@

from pt_snap_cli import __version__
from pt_snap_cli.context import Context, DatabaseNotFoundError, SchemaVersionError
from pt_snap_cli.core.dataset_resolver import DatasetResolver
from pt_snap_cli.core.dataset_resolver import DatasetResolver, ResolvedDataset
from pt_snap_cli.core.errors import DatabaseMissingError, DatabaseSchemaError, ImportMetadataError
from pt_snap_cli.core.import_metadata_contract import (
metadata_from_mapping,
Expand Down Expand Up @@ -83,22 +83,36 @@ def build_metadata(
completed_at=datetime.now(timezone.utc).isoformat(),
)

def inspect(self, db_path: Path | str) -> MetadataInspection:
dataset = DatasetResolver().inspect(db_path)
def inspect(
self, db_path: Path | str, *, _resolved: ResolvedDataset | None = None
) -> MetadataInspection:
dataset = _resolved if _resolved is not None else DatasetResolver().inspect(db_path)
path = Path(db_path).expanduser().resolve()
if dataset is not None:
if path not in (dataset.root, dataset.root / "manifest.json"):
raise DatabaseSchemaError("Dataset resolution does not match metadata path")
if isinstance(dataset.validation.manifest, NativeManifest):
dataset.require_unchanged()
return MetadataInspection(path, "available", dataset.validation.manifest.metadata)
# External P0 metadata remains optional. Only a recognized complete
# pt-snap manifest attestation can provide dataset-wide metadata.
with (dataset.root / "manifest.json").open(encoding="utf-8") as source:
raw = json.load(source)
if "ptSnap" in raw:
try:
owned = inspect_owned_cache(dataset.root)
except (ValueError, KeyError, TypeError, OSError, sqlite3.DatabaseError):
owned = inspect_owned_cache(dataset.root, _resolved=dataset)
except (
ValueError,
KeyError,
TypeError,
OSError,
sqlite3.DatabaseError,
DatabaseSchemaError,
):
dataset.require_unchanged()
return MetadataInspection(path, "invalid", reason="metadata_invalid")
return MetadataInspection(path, "available", owned.metadata)
dataset.require_unchanged()
return MetadataInspection(path, "unavailable", reason="metadata_missing")
try:
context = Context(path)
Expand Down
Loading
Loading