diff --git a/README.md b/README.md index 8ac0dc0..c84c9ad 100644 --- a/README.md +++ b/README.md @@ -56,6 +56,16 @@ Complete compatibility-v1 sharded artifacts can be focused by directory or `manifest.json`, without the archived pickle. See [dataset focus and addressing](docs/en/sharded-snapshotdb.md#complete-dataset-focus-and-addressing-p1) for read-only overview, event routing, `--slice`, and explicit cross-slice limits. +Opt into a **native, not msinsight-compatible** dataset with +`pt-snap import snapshot.pkl --events-per-slice 50000 --output-dir captures --json`. +The output is `captures/snapshot.pkl.pt-snap-native-v2/`; default standalone import +is unchanged. Reuse validates all shards and content/options/semantic versions. +Nonmatching recognized targets require `--force`; unknown targets are preserved. +Publication/focus failures are compensated, with explicit recovery evidence if +rollback fails (force is not a single crash-atomic swap). See the +[native import guide](docs/en/quickstart.md#optional-import-a-native-sharded-dataset). +Only bounded dataset `event` queries are currently supported. + ## Commands | Command | Description | @@ -102,7 +112,7 @@ See the [documentation index](docs/README.md) for all English and Chinese guides | Splitting snapshots | [Splitting Snapshots](docs/en/splitting.md) | | Agent skills | [Agent Skills](docs/en/skills.md) | | Database format | [SnapshotDB Schema](docs/en/database.md) | -| Sharded contract and internal replay (P0/P1; no CLI exporter) | [Sharded SnapshotDB protocol](docs/en/sharded-snapshotdb.md) | +| Sharded protocols, native import and compatibility limits | [Sharded SnapshotDB protocol](docs/en/sharded-snapshotdb.md) | | Python API | [SnapshotAnalyzer API](docs/en/snapshot-analyzer-api.md) | | Result mapping utility | [ResultMapper API](docs/en/result-mapper-api.md) | diff --git a/README_zh.md b/README_zh.md index 7c814e4..63f29f3 100644 --- a/README_zh.md +++ b/README_zh.md @@ -51,6 +51,13 @@ pt-snap split snapshot.pkl --slices 4 --output snapshot-slices 只读 overview、事件路由、`--slice` 和跨片限制见 [完整数据集 focus 与寻址](docs/zh/sharded-snapshotdb.md#完整数据集-focus-与寻址p1)。 +用 `pt-snap import snapshot.pkl --events-per-slice 50000 --output-dir captures --json` +显式生成**原生、非 msinsight 兼容**数据集 `captures/snapshot.pkl.pt-snap-native-v2/`。 +默认单库导入不变;复用校验全部片及内容/选项/语义版本。不匹配的已识别目标须 +`--force`,未知目标保留。发布/focus 失败会补偿,回滚自身失败则保留并报告恢复证据 +(force 不是单次崩溃原子 swap)。详见 +[原生分库导入](docs/zh/quickstart.md#可选导入原生分库数据集);当前仅支持有界数据集 `event` 查询。 + ## 命令 | 命令 | 说明 | @@ -95,7 +102,7 @@ helper 会先路由采集,无需先安装 CLI。分析仍需要 CLI 和 Snapsh | 拆分快照 | [拆分快照](docs/zh/splitting.md) | | Agent skill | [Agent Skills](docs/zh/skills.md) | | 数据库格式 | [SnapshotDB Schema](docs/zh/database.md) | -| 分片合同与内部回放(P0/P1,无 CLI 导出器) | [分片 SnapshotDB 协议](docs/zh/sharded-snapshotdb.md) | +| 分片协议、原生导入与兼容边界 | [分片 SnapshotDB 协议](docs/zh/sharded-snapshotdb.md) | | Python API | [SnapshotAnalyzer API](docs/zh/snapshot-analyzer-api.md) | | 结果映射工具 | [ResultMapper API](docs/zh/result-mapper-api.md) | diff --git a/docs/en/database.md b/docs/en/database.md index 14c32cc..206c908 100644 --- a/docs/en/database.md +++ b/docs/en/database.md @@ -386,3 +386,23 @@ CREATE TABLE pt_snap_metadata ( Legacy and externally produced compatible databases may omit this table. They remain queryable, but `pt-snap metadata` reports their metadata as unavailable. + +### Published native datasets + +`import --events-per-slice N` publishes a distinct `pt-snap-native-v2` manifest, +not a compatibility-v1 manifest or another standalone DB layout. Each finalized +shard uses the existing v2 tables plus `pt_snap_block_reference` (version 1), with +`blockId INTEGER PRIMARY KEY`, `stream INTEGER NOT NULL`, `allocCallstack TEXT`, +`freeCallstack TEXT`. These are text lifecycle sources, not structured frames or +an automatic cross-shard JOIN. `pt_snap_metadata` is required and identical in all +members; native manifest identity additionally versions source/device/capacity, +format/manifest/import/extension contracts, independently of CLI release version. + +Real counts use positions per device, preserving original sparse IDs and native +OOM/workspace events. Negative boundaries do not consume capacity. Reader +validation includes full-member hashes and finalized identity/reference consistency +before bounded `event` addressing; it never repairs or migrates files. Aggregate +and lifecycle dataset queries remain unsupported. See the separate +[native manifest protocol](sharded-snapshotdb.md#published-native-dataset-p2) and +[transaction/recovery guide](quickstart.md#optional-import-a-native-sharded-dataset). +P0 compatibility-v1 table order, inline callstack layout and reject rules are unchanged. diff --git a/docs/en/focus-management.md b/docs/en/focus-management.md index 4775e99..386cc70 100644 --- a/docs/en/focus-management.md +++ b/docs/en/focus-management.md @@ -112,6 +112,33 @@ ignore rules automatically. } ``` +## Complete Dataset Focus and Import Compensation + +A complete compatibility-v1 dataset or published `pt-snap-native-v2` directory/ +`manifest.json` can be focused with the same precedence and device rules: + +```bash +pt-snap focus captures/snapshot.pkl.pt-snap-native-v2 --device 0 --json +pt-snap query --template-use event --slice 0 --json +``` + +Native layout is `v2`; compatibility layout is `v1`. All declared members are +validated before writing focus, with no artifact repairs or layout persistence. +Native sparse IDs are addressed as original IDs, not positional counts. Only +bounded `event` queries are supported through dataset focus; aggregate/lifecycle +and cross-slice requests fail explicitly. API session focus is read-only and +`SnapshotAnalyzer` has no import/split facade. + +Import writes project focus in the **current directory**, not the output directory. +`--no-focus` skips it. A native dataset import focus failure, including mutation before an error, +restores the old artifact plus exact prior focus bytes and in-memory config state +(or prior absence) on ordinary compensation. If restoration fails, the error +identifies the retained `.focus.pt-snap-*.recovery` checkpoint and/or dataset +recovery directory. An empty checkpoint with `prior focus existed: False` records +prior absence, not a valid focus JSON. Preserve evidence for manual recovery; +no unconditional I/O/crash atomicity or concurrent-writer lock is promised. +See [native import](quickstart.md#optional-import-a-native-sharded-dataset). + ## Error Handling **No focus configured:** diff --git a/docs/en/quickstart.md b/docs/en/quickstart.md index ba08d6b..39ff874 100644 --- a/docs/en/quickstart.md +++ b/docs/en/quickstart.md @@ -49,6 +49,58 @@ copying raw events directly. The resulting SnapshotDB records event-by-event the supported input for `pt-snap query` and `pt-snap report`. This replay behavior is part of the command workflow; snapshot runtime Python modules are not a public API. +### Optional: Import a Native Sharded Dataset + +```bash +pt-snap import snapshot.pkl --events-per-slice 50000 --output-dir captures --json +pt-snap overview captures/snapshot.pkl.pt-snap-native-v2 --json +pt-snap query --template-use event --slice 0 --json +``` + +Without `--events-per-slice`, import still produces the existing single DB. With +it, the output is `/.pt-snap-native-v2/`, or the +same directory beside the source if `--output-dir` is omitted. `--format +pt-snap-native-v2` is optional in this mode; `--format single-db` forbids a capacity. +`--format compatibility-v1` and `msinsight` are reserved and explicitly rejected: +native output is **not** an msinsight-compatible export. + +Capacity counts real chronological positions **per device**, not `max(id)+1` or +negative synthetic boundaries. Original sparse/nonzero IDs, OOM and workspace +events remain native. `--device` selects one event-bearing device; otherwise all +such devices are imported. Empty/static-only and unselected device positions are +reported in `omitted_devices`; selecting an empty device or an all-empty input +fails. Sharding does not bound pickle loading, registry/block rows or peak RSS. + +Reuse requires a validated complete manifest and **every** closed member, with +matching source SHA256, device selection, capacity, native format, manifest, +semantic import, extension and metadata/layout contracts. CLI release version +changes alone do not rebuild. Native identity is separate from future compatible +format identities and from msinsight's salted `cacheHash`. JSON adds `dataset_path`, +`devices`, `slice_count`, `format`, `omitted_devices`, `reused` and `rebuilt`; +`db_path` denotes the dataset directory in this mode. + +A matching target is reused. A different/corrupt recognized native artifact is +preserved unless `--force` explicitly requests replacement. Unknown directories, +extra unrelated files, malformed manifests and symlink targets are never adopted +or deleted, even with force: choose a new `--output-dir`. All devices/shards are +staged beside the target, finalized, closed, metadata-attested and validated; the +source is hashed again before publication. No partial shard is a reusable cache. + +New publication uses an exclusive directory rename. Force replacement first +renames the old artifact into an owned recovery directory, then publishes the +new one; **there is a transient path gap, not a single atomic swap**. Requested +project focus is part of compensation, including a focus write that mutates +before raising: ordinary failure restores old artifact and exact old focus +bytes/state (or previous absence). `--no-focus` skips focus entirely. If rollback +itself fails, recovery paths are preserved and explicitly reported. Do not delete +them before inspecting the evidence. Arbitrary I/O failures, concurrent writers +and crashes do not have unconditional atomic restoration guarantees. + +Dataset focus currently supports only bounded `event` addressing; it does not +silently run peak/lifecycle/aggregate queries over one shard. See the +[native manifest contract](sharded-snapshotdb.md#published-native-dataset-p2). +`SnapshotAnalyzer` remains analysis-only, without import/split methods. + ### Optional: Split a Snapshot Use `pt-snap split` when you need smaller, independently replayable files. Split diff --git a/docs/en/sharded-snapshotdb.md b/docs/en/sharded-snapshotdb.md index 65a9d5b..c26ee75 100644 --- a/docs/en/sharded-snapshotdb.md +++ b/docs/en/sharded-snapshotdb.md @@ -1,4 +1,4 @@ -# Sharded SnapshotDB protocol (P0, compatibility v1) +# Sharded SnapshotDB protocols (compatibility v1 and native v2) [中文](../zh/sharded-snapshotdb.md) | English @@ -247,8 +247,64 @@ mode; standalone DB behavior is unchanged. This is read-only inspection, not a filesystem sandbox or a concurrent producer lock. Native-v2 replay's private staging directory has **no published manifest** and is rejected as a dataset; its individual DBs remain readable via the existing single-DB entry. A future compatibility-v1 producer can use this same reader -without producer-specific metadata. No native-v2 manifest protocol, exporter, -cache publication, migration, or cross-slice lifecycle reconstruction is invented. +without producer-specific metadata. The P1 reader does not create a native manifest; publication requires the explicit +P2 native protocol below. Compatibility export and cross-slice lifecycle execution +remain separate work. In particular, the fixed original compatibility producer can +leave finalized, sidecar-free WAL headers, which this reader still rejects. Transport +support for those artifacts is a compatibility prerequisite, not passed upstream/GUI +interoperability; analysis never rewrites them. + +## Published Native Dataset (P2) + +This is an **original, separately named and versioned** pt-snap protocol, not a +relabeled P0 artifact. `ImportService`/CLI can publish the internal native replay +result as `/.pt-snap-native-v2/`. The root must +contain exactly `manifest.json` plus the declared `device_` directories; each +device directory contains exactly its canonical `slice_.db` members. + +Required native manifest fields: + +| Field | Contract | +| --- | --- | +| `format` | Exactly `pt-snap-native-v2`, independently dispatched from compatibility-v1 | +| `schemaVersion`, `status` | Native manifest version `1`, only `complete` | +| `sourceFile` | Nonempty source basename matching `metadata.source_name`; never opened by analysis | +| `identity` | Exact fields `sourceSha256`, `requestedDevice`, `eventsPerSlice`, `outputFormat`, `manifestContractVersion`, `formatContractVersion`, `importContractVersion`, `extensionContractVersion`, `metadataSchemaVersion`, `importFormatVersion` | +| `metadata` | Complete existing `ImportMetadata` object, identical to the single row in every shard | +| `omittedDevices` | Distinct unselected/empty/static-only source device positions, disjoint from selected devices | +| `devices` | Canonical decimal keys; positive `eventCount`, `sliceCount`, all `readySlices` in order, ordered `slices` | +| Each slice | `index`, `startPosition`, `endPosition`, `startEventId`, `endEventId`, canonical relative `file`, `ready: true`, closed-member `sha256` | + +All semantic contract versions start at `1`; native metadata schema is `1` and DB +import format is `2`. `requestedDevice` is null (all event-bearing devices) or one +nonnegative integer. SHA256 strings are lowercase 64-digit hex. CLI +`metadata.importer_version` is informational, **not** a cache invalidation axis. +Native and future compatible identities are separately versioned; no native +`cacheHash` is generated or accepted as an identity substitute. + +Positions continuously cover `0..eventCount-1` per device, with every nonfinal +slice full and the final slice at most capacity. ID ranges are inclusive original +chronological bounds and may be sparse/nonzero; real count comes from positions +and actual rows, never `max(id)+1`. Negative synthetic rows are counted separately. +Each member has exactly its native-v2 trace/block device tables, `dictionary`, +`callstack`, `pt_snap_metadata`, and finalized `pt_snap_block_reference` v1. Native +trace columns match the standalone v2 schema (`callstackId`, not inline text), +with native OOM action 8 retained. Block IDs/lifetimes and text references must be +finalized, consistent across slices and refer to original IDs in that device. +Unknown extension declarations/tables do not grant structured-frame capability. + +The native resolver validates all member hashes, actual schemas/dictionaries, +real count/ranges, metadata and finalized references, retaining the P1 pre-open +alias/sidecar/persistent-WAL checks and bounded borrowed-cache ownership. Focus, +metadata, overview and single-slice `event` addressing work without the source +pickle. Native metadata is available dataset-wide; compatibility metadata remains +unavailable. No migration, repair, cross-slice aggregation/lifecycle execution, +msinsight exporter, salted hash policy or GUI acceptance is implied. The P0 +schema and reject policies above remain unchanged. + +Publication and force/focus compensation are described in the +[import guide](quickstart.md#optional-import-a-native-sharded-dataset). A complete +manifest becomes visible only with all finalized members, never one ready shard. ## Internal continuous replay (P1) diff --git a/docs/zh/database.md b/docs/zh/database.md index 94fdec2..52e96d2 100644 --- a/docs/zh/database.md +++ b/docs/zh/database.md @@ -368,3 +368,19 @@ CREATE TABLE pt_snap_metadata ( 旧版或外部生成的兼容数据库可以不包含此表,仍可正常查询;`pt-snap metadata` 会将其 metadata 状态报告为 unavailable。 + +### 已发布原生数据集 + +`import --events-per-slice N` 发布独立 `pt-snap-native-v2` manifest,不是兼容 v1 +manifest,也不改变单库布局。各最终化片使用已有 v2 表及 `pt_snap_block_reference` +(版本 1):`blockId INTEGER PRIMARY KEY`、`stream INTEGER NOT NULL`、 +`allocCallstack TEXT`、`freeCallstack TEXT`。它们记录文本生命周期来源,不是结构化 +frames 或自动跨片 JOIN。每片必需且一致的 `pt_snap_metadata` 之外,原生 manifest +身份另包含源/设备/容量、格式/manifest/导入/扩展合同版本,与 CLI 发布版本分离。 + +真实计数按各设备位置,保留原始稀疏 ID、OOM/workspace;负边界不占容量。 +reader 在有界 `event` 寻址前校验完整成员哈希、最终化身份/引用一致性,不修复、不迁移。 +数据集聚合/生命周期查询仍不支持。详见独立 +[原生 manifest 协议](sharded-snapshotdb.md#已发布原生数据集p2)及 +[事务恢复指南](quickstart.md#可选导入原生分库数据集)。 +P0 兼容 v1 表序、内联 callstack 布局和拒绝规则不变。 diff --git a/docs/zh/focus-management.md b/docs/zh/focus-management.md index e67d323..d56324b 100644 --- a/docs/zh/focus-management.md +++ b/docs/zh/focus-management.md @@ -110,6 +110,28 @@ pt-snap config --json # 同一组操作用机器可读 JSON 输出 } ``` +## 完整数据集 focus 与导入补偿 + +完整兼容 v1 数据集或已发布 `pt-snap-native-v2` 目录/`manifest.json` 可以按相同的 +优先级和设备规则 focus: + +```bash +pt-snap focus captures/snapshot.pkl.pt-snap-native-v2 --device 0 --json +pt-snap query --template-use event --slice 0 --json +``` + +原生布局为 `v2`,兼容布局为 `v1`;写 focus 前校验全部声明成员,不修复产物,不保存 +布局信息。原生稀疏 ID 按原始 ID 寻址,不当成位置计数。数据集 focus 只支持有界 +`event`,聚合/生命周期及跨片请求明确失败。API 会话 focus 只读,`SnapshotAnalyzer` +不承担 import/split。 + +导入把项目 focus 写入**当前目录**,不是输出目录;`--no-focus` 跳过。原生数据集导入 focus 失败 +(包括先修改再抛错)在普通补偿下恢复旧产物与旧 focus 原始字节及内存 config 状态, +或者恢复原先不存在。恢复失败时,错误指明保留的 `.focus.pt-snap-*.recovery` checkpoint +及/或数据集恢复目录。若 checkpoint 为空且 `prior focus existed: False`,它记录原先 +不存在,不是有效 focus JSON。请保留证据供人工恢复;不承诺任意 I/O/崩溃原子性或并发 +writer 锁。详见[原生导入](quickstart.md#可选导入原生分库数据集)。 + ## 错误处理 **未设置焦点时查询:** diff --git a/docs/zh/quickstart.md b/docs/zh/quickstart.md index 9a82241..788be60 100644 --- a/docs/zh/quickstart.md +++ b/docs/zh/quickstart.md @@ -44,6 +44,46 @@ SnapshotDB 会记录每个事件后的 `allocated`、`active`、`reserved` 总 生命周期,供 `pt-snap query` 与 `pt-snap report` 使用。回放是命令工作流的一部分; 快照运行时的 Python 模块不是公开 API。 +### 可选:导入原生分库数据集 + +```bash +pt-snap import snapshot.pkl --events-per-slice 50000 --output-dir captures --json +pt-snap overview captures/snapshot.pkl.pt-snap-native-v2 --json +pt-snap query --template-use event --slice 0 --json +``` + +不指定 `--events-per-slice` 时仍生成已有单 DB;指定后生成 +`/<完整输入文件名>.pt-snap-native-v2/`,省略 `--output-dir` 则放在源文件旁。 +此模式可显式指定 `--format pt-snap-native-v2`;`--format single-db` 不允许容量参数。 +`--format compatibility-v1` 和 `msinsight` 是保留边界并明确拒绝:原生产物**不是** +msinsight 兼容导出。 + +容量按**各设备**的真实时间顺序位置计数,不使用 `max(id)+1`,不计负 ID 合成边界。 +保留原始稀疏/非零 ID、OOM 和 workspace 事件。`--device` 选择一个有事件设备,默认 +导入全部有事件设备;空/仅静态及未选设备位置由 `omitted_devices` 披露。选择空设备或 +全部为空的输入会失败。分库不限制 pickle 加载、registry/block 行数或峰值 RSS。 + +缓存复用须校验完整 manifest 与**全部**已关闭成员,并匹配源 SHA256、设备选择、容量、 +原生格式、manifest、导入语义、扩展及 metadata/布局合同版本。仅 CLI 发布版本变化不会 +重建;原生身份与后续兼容格式及 msinsight salted `cacheHash` 分离。JSON 增加 +`dataset_path`、`devices`、`slice_count`、`format`、`omitted_devices`、`reused` 和 +`rebuilt`;此模式下 `db_path` 为数据集目录。 + +匹配目标可复用;不同或成员损坏的已识别原生数据集会保留,除非显式 `--force` 替换。 +未知目录、无关额外文件、非法 manifest 和符号链接目标即使 force 也不会被接管或删除, +请换一个 `--output-dir`。所有设备/分片在目标同级暂存,完成回填、关闭、metadata 写入与 +校验,并在发布前复核源哈希;部分就绪片不构成缓存。 + +新目标使用排他的目录 rename 发布。force 先把旧产物移入所有权受检的恢复目录,再发布 +新产物;**路径有短暂空档,不是单次原子 swap**。请求的项目 focus 纳入补偿,写入已修改 +文件后才抛错也要处理:普通失败恢复旧产物及旧 focus 原始字节/状态(或原先不存在)。 +`--no-focus` 完全跳过 focus。回滚自身失败会保留并明确报告恢复路径,检查证据前不要删除。 +不保证任意 I/O 故障、并发 writer 或崩溃下无条件原子恢复。 + +数据集 focus 当前只支持有界 `event` 寻址,不会静默在一片上运行峰值/生命周期/聚合。 +详见[原生 manifest 合同](sharded-snapshotdb.md#已发布原生数据集p2)。 +`SnapshotAnalyzer` 仍只负责分析,不新增 import/split 方法。 + ### 可选:拆分快照 如需生成更小、可独立回放的文件,可使用 `pt-snap split`。该命令不会读取或修改 focus: diff --git a/docs/zh/sharded-snapshotdb.md b/docs/zh/sharded-snapshotdb.md index 9c0159c..e8cae06 100644 --- a/docs/zh/sharded-snapshotdb.md +++ b/docs/zh/sharded-snapshotdb.md @@ -1,4 +1,4 @@ -# 分片 SnapshotDB 协议(P0,兼容 v1) +# 分片 SnapshotDB 协议(兼容 v1 与原生 v2) 中文 | [English](../en/sharded-snapshotdb.md) @@ -201,8 +201,52 @@ WAL/journal/SHM sidecar(包含悬空符号链接)。即使 checkpoint 已移 模式仍被拒绝:`mode=ro` 也可能重新创建它们。reader 不修复、不执行 checkpoint、不删除 sidecar、不改变 journal mode;单库行为不变。这是只读检查,不是文件系统沙箱或并发生产端锁。native-v2 replay 的私有 staging **没有已发布 manifest**,不能作为数据集;其中单个 DB 仍由已有单库入口读取。 -后续兼容 v1 生产端可直接使用同一 reader,不依赖生产端专属 metadata。本项不虚构 -native-v2 manifest 协议,不新增导出、缓存发布、迁移或跨片生命周期重建。 +后续兼容 v1 生产端可直接使用同一 reader,不依赖生产端专属 metadata。P1 reader 不创建原生 manifest;发布须采用下述显式 P2 原生协议。兼容导出与跨片 +生命周期执行仍属独立工作。特别是固定版本原始兼容生产端可能留下最终化、无 sidecar +的 WAL header,本 reader 仍拒绝;支持该类产物的只读传输是兼容前提,不代表上游/GUI +互通已验收,分析不会改写它们。 + +## 已发布原生数据集(P2) + +这是 pt-snap **原创、独立命名和版本化**的协议,不是重标记的 P0 产物。ImportService/CLI +将内部原生回放结果发布至 `/<完整源文件名>.pt-snap-native-v2/`。根目录只能 +包含 `manifest.json` 与声明的 `device_` 目录;各设备目录只能有规范的 +`slice_.db` 成员。 + +原生 manifest 必需字段: + +| 字段 | 合同 | +| --- | --- | +| `format` | 严格为 `pt-snap-native-v2`,与 compatibility-v1 独立分派 | +| `schemaVersion`, `status` | 原生 manifest 版本 `1`,只允许 `complete` | +| `sourceFile` | 非空源 basename,与 `metadata.source_name` 一致,分析不打开源文件 | +| `identity` | 精确字段 `sourceSha256`、`requestedDevice`、`eventsPerSlice`、`outputFormat`、`manifestContractVersion`、`formatContractVersion`、`importContractVersion`、`extensionContractVersion`、`metadataSchemaVersion`、`importFormatVersion` | +| `metadata` | 完整已有 `ImportMetadata` 对象,与每片唯一 metadata 行一致 | +| `omittedDevices` | 不重复的未选/空/仅静态源设备位置,与所选设备不相交 | +| `devices` | 规范十进制 key,正 `eventCount/sliceCount`,全部按序 `readySlices` 及按序 `slices` | +| 各 slice | `index`、`startPosition/endPosition`、`startEventId/endEventId`、规范相对 `file`、`ready: true`、已关闭成员 `sha256` | + +所有语义合同版本初始为 `1`;原生 metadata schema 为 `1`、DB import format 为 `2`。 +`requestedDevice` 为 null(全部有事件设备)或一个非负整数;SHA256 为 64 位小写十六进制。 +CLI `metadata.importer_version` 仅提供信息,**不是**缓存失效轴。原生与后续兼容格式身份 +独立版本化,不生成原生 `cacheHash`,也不接受它替代缓存身份。 + +位置在各设备内连续覆盖 `0..eventCount-1`,末片之前全部满容量,末片至多容量个事件。 +ID 区间为原始时间顺序闭边界,可稀疏/非零;真实计数来自位置和实际行,不是 +`max(id)+1`。负 ID 合成行单独计数。各成员须有本设备的原生 v2 trace/block 实际表、 +`dictionary`、`callstack`、`pt_snap_metadata` 与最终化 `pt_snap_block_reference` v1。 +trace 列与单库 v2 一致(`callstackId`,非内联文本),保留 OOM action 8。block 身份/ +生命周期和文本引用须最终化、跨片一致,并引用原设备真实 ID。未知扩展声明/表不会 +提升结构化 frame 能力。 + +原生 resolver 校验全部成员哈希、实际 schema/dictionary、真实计数/范围、metadata 和 +最终化引用,保留 P1 打开前的别名/sidecar/持久 WAL 检查及有界借用 cache 所有权。 +无需源 pickle 即可 focus、metadata、overview 和单片 `event` 寻址。原生数据集 metadata +为 available,兼容数据集仍为 unavailable。不隐含迁移、修复、跨片聚合/生命周期执行、 +msinsight 导出器、salted hash 策略或 GUI 验收;以上 P0 schema 和拒绝规则保持不变。 + +发布、force/focus 补偿见[导入指南](quickstart.md#可选导入原生分库数据集)。完整 manifest +只能与全部最终化成员同时可见,单个 ready 片不构成完整产物。 ## 内部持续回放(P1) diff --git a/src/pt_snap_cli/cli.py b/src/pt_snap_cli/cli.py index fcb2a27..648533c 100644 --- a/src/pt_snap_cli/cli.py +++ b/src/pt_snap_cli/cli.py @@ -149,6 +149,18 @@ def _import_json(result: ImportResult) -> dict[str, JsonValue]: metadata=asdict(result.metadata), focus_state=focus_state, focus_source=result.focus_state.source if result.focus_state is not None else None, + **( + { + "dataset_path": str(result.dataset_path), + "devices": list(result.devices), + "slice_count": result.slice_count, + "format": result.format, + "omitted_devices": list(result.omitted_devices), + "rebuilt": not result.reused, + } + if result.dataset_path is not None + else {} + ), ) @@ -378,6 +390,16 @@ def import_snapshot( device: Annotated[int | None, typer.Option("--device", "-d")] = None, no_focus: Annotated[bool, typer.Option("--no-focus", help="Skip focus update")] = False, force: Annotated[bool, typer.Option("--force", help="Rebuild even when cache matches")] = False, + events_per_slice: Annotated[ + int | None, + typer.Option( + "--events-per-slice", help="Real events per device shard; opt into a native dataset" + ), + ] = None, + output_format: Annotated[ + str | None, + typer.Option("--format", help="single-db or pt-snap-native-v2 (compatibility-v1 reserved)"), + ] = None, json_output: Annotated[bool, _json_flag()] = False, ) -> None: """Import a PyTorch memory snapshot into a SQLite database.""" @@ -389,6 +411,8 @@ def import_snapshot( device=device, set_focus=not no_focus, force=force, + events_per_slice=events_per_slice, + format=output_format, ), json_output=json_output, ) @@ -396,6 +420,7 @@ def import_snapshot( ImportToolMissingError, ImportExecutionError, SnapshotFileInvalidError, + InvalidParameterError, ) as e: _error_from_exc(e) @@ -405,6 +430,10 @@ def import_snapshot( action = "Reused" if result.reused else "Imported" typer.echo(f"{action}: {result.db_path}") + if result.dataset_path is not None: + typer.echo( + f"Format: {result.format}; devices: {list(result.devices)}; slices: {result.slice_count}" + ) if result.cache_miss_reason is not None: typer.echo(f"Cache miss: {result.cache_miss_reason}") if result.focus_state is not None and result.focus_state.focus_file is not None: diff --git a/src/pt_snap_cli/core/dataset_files.py b/src/pt_snap_cli/core/dataset_files.py new file mode 100644 index 0000000..55e7153 --- /dev/null +++ b/src/pt_snap_cli/core/dataset_files.py @@ -0,0 +1,35 @@ +"""Shared pre-open dataset checks. No repair, migration, WAL changes or SQLite opens.""" + +from __future__ import annotations + +import hashlib +from pathlib import Path + +from .errors import DatabaseSchemaError + + +def require_readonly_member(root: Path, member: Path) -> None: + parts = member.relative_to(root).parts + for index in range(1, len(parts) + 1): + if root.joinpath(*parts[:index]).is_symlink(): + raise DatabaseSchemaError(f"Dataset member must not use symlink paths: {member}") + if member.resolve() != member or not member.is_file(): + raise DatabaseSchemaError(f"Dataset member is missing or not a canonical file: {member}") + for suffix in ("-wal", "-journal", "-shm"): + sidecar = Path(str(member) + suffix) + if sidecar.exists() or sidecar.is_symlink(): + raise DatabaseSchemaError(f"Dataset member has a live SQLite sidecar: {member}") + with member.open("rb") as source: + header = source.read(20) + # Retain #200's persistent-WAL rejection, including sidecar-free checkpointed + # WAL. Accepting fixed-upstream compatibility producers is a separate boundary. + if header[:16] == b"SQLite format 3\x00" and 2 in header[18:20]: + raise DatabaseSchemaError(f"Dataset member uses persistent WAL mode: {member}") + + +def hash_file(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as source: + for chunk in iter(lambda: source.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() diff --git a/src/pt_snap_cli/core/dataset_import_backend.py b/src/pt_snap_cli/core/dataset_import_backend.py new file mode 100644 index 0000000..ab786d6 --- /dev/null +++ b/src/pt_snap_cli/core/dataset_import_backend.py @@ -0,0 +1,219 @@ +"""Native dataset publication: exclusive renames, explicit compensation, no crash guarantee.""" + +from __future__ import annotations + +import shutil +import tempfile +from collections.abc import Callable +from dataclasses import dataclass +from pathlib import Path + +from .dataset_contract import DatasetContractError +from .dataset_resolver import _hash_file +from .errors import ImportExecutionError +from .native_dataset_contract import NativeManifest, read_native_manifest, require_native_members +from .sharded_replay_service import ShardedReplayResult, ShardedReplayService +from .split_service import SplitService + +DirectoryIdentity = tuple[int, int, int | None] + + +@dataclass(frozen=True) +class TargetState: + device: int + inode: int + files: tuple[tuple[str, str], ...] + + +class DatasetImportBackend: + def __init__(self, replay: ShardedReplayService | None = None): + self.replay = replay or ShardedReplayService() + + @staticmethod + def target_path(source: Path, output_dir: Path) -> Path: + return output_dir / f"{source.name}.pt-snap-native-v2" + + @staticmethod + def inspect_target(target: Path) -> tuple[NativeManifest, TargetState] | None: + if not target.exists() and not target.is_symlink(): + return None + try: + if target.resolve() != target or target.is_symlink() or not target.is_dir(): + raise ValueError("Target must be a canonical non-symlink native dataset directory") + manifest = read_native_manifest(target) + require_native_members(target, manifest) + status = target.lstat() + files = tuple( + (name, _hash_file(target / name)) + for name in sorted( + ["manifest.json", *(s.file for d in manifest.devices for s in d.slices)] + ) + ) + return manifest, TargetState(status.st_dev, status.st_ino, files) + except (DatasetContractError, OSError, ValueError) as exc: + raise ImportExecutionError( + f"Existing target is not a recognized native dataset; preserved: {exc}" + ) from exc + + @staticmethod + def _verify(path: Path, identity: DirectoryIdentity) -> None: + if path.resolve() != path: + raise OSError(f"Noncanonical or substituted directory: {path}") + SplitService._verify_stage_identity(path, identity) + + @classmethod + def _remove(cls, path: Path, identity: DirectoryIdentity) -> None: + cls._verify(path, identity) + # shutil.rmtree does not traverse child symlink targets. The root is held + # by an inode-checked descriptor from secure task allocation/recognition. + shutil.rmtree(path) + + @classmethod + def _move(cls, source: Path, destination: Path, identity: DirectoryIdentity) -> None: + cls._verify(source, identity) + if destination.parent.resolve() != destination.parent or destination.is_symlink(): + raise OSError(f"Unsafe publication destination: {destination}") + SplitService._publish_directory(source, destination) + + @classmethod + def _matches(cls, path: Path, identity: DirectoryIdentity) -> bool: + try: + cls._verify(path, identity) + return True + except OSError: + return False + + def dump_to_dataset( + self, + source: Path, + target: Path, + *, + capacity: int, + device: int | None, + expected: TargetState | None, + finalize: Callable[[ShardedReplayResult], None], + post_publish: Callable[[Path], None] | None = None, + pre_publish: Callable[[], None] | None = None, + ) -> Path: + if ( + target.resolve() != target + or target.parent.resolve() != target.parent + or target.is_symlink() + ): + raise ImportExecutionError("Dataset output must be canonical and non-symlink.") + stage: Path | None = None + recovery: Path | None = None + stage_id: DirectoryIdentity | None = None + recovery_id: DirectoryIdentity | None = None + old_id: DirectoryIdentity | None = None + new_id: DirectoryIdentity | None = None + preserve = False + try: + SplitService._preflight_publication() + target.parent.mkdir(parents=True, exist_ok=True) + current = self.inspect_target(target) + if (current[1] if current else None) != expected: + raise OSError("Dataset destination changed before staging; preserved") + if expected is not None: + old_id = SplitService._stage_identity(target) + stage = Path(tempfile.mkdtemp(dir=target.parent, prefix=f".{target.name}.stage-")) + stage_id = SplitService._stage_identity(stage) + artifact = stage / "artifact" + result = self.replay.stage(source, artifact, events_per_slice=capacity, device=device) + self._verify(stage, stage_id) + new_id = SplitService._stage_identity(artifact) + finalize(result) + self._verify(artifact, new_id) + current = self.inspect_target(target) if expected is not None else None + if (current[1] if current is not None else None) != expected: + raise OSError("Dataset destination changed during import; preserved") + if pre_publish is not None: + pre_publish() + try: + if old_id is not None: + self._verify(target, old_id) + recovery = Path( + tempfile.mkdtemp(dir=target.parent, prefix=f".{target.name}.recovery-") + ) + recovery_id = SplitService._stage_identity(recovery) + self._move(target, recovery / "previous", old_id) + self._move(artifact, target, new_id) + if post_publish is not None: + post_publish(target) + except BaseException as publish_error: + try: + # Detect successful rename BEFORE an injected call raises. + # Never remove an independently created/substituted target. + if self._matches(target, new_id): + self._remove(target, new_id) + if old_id is not None: + if not self._matches(target, old_id): + if ( + recovery is None + or recovery_id is None + or not self._matches(recovery / "previous", old_id) + ): + raise OSError( + "Previous artifact is missing or substituted; recovery cannot be trusted" + ) + self._verify(recovery, recovery_id) + previous = self.inspect_target(recovery / "previous") + if previous is None or previous[1] != expected: + raise OSError( + "Recovery artifact changed; preserve evidence instead of restoring unknown data" + ) + try: + self._move(recovery / "previous", target, old_id) + except BaseException: + # A late exception may follow a successful restore. + restored = self.inspect_target(target) + if ( + not self._matches(target, old_id) + or restored is None + or restored[1] != expected + ): + raise + restored = self.inspect_target(target) + if restored is None or restored[1] != expected: + raise OSError( + "Previous artifact was not restored exactly; preserve recovery evidence" + ) + except BaseException as rollback_error: + preserve = True + raise ImportExecutionError( + f"Dataset publication/focus failed; directory rollback also failed: {rollback_error}. " + f"Recovery directory: {recovery}; staging evidence: {stage}." + ) from publish_error + raise + # The transaction is committed only after the focus callback returns. + if recovery is not None and old_id is not None: + try: + previous = self.inspect_target(recovery / "previous") + if previous is None or previous[1] != expected: + raise OSError("Recovery artifact changed; do not delete unknown contents") + self._remove(recovery / "previous", old_id) + except Exception as exc: + preserve = True + raise ImportExecutionError( + f"Dataset publication committed but recovery cleanup failed; preserved evidence at {recovery}: {exc}" + ) from exc + return target + except ImportExecutionError: + raise + except Exception as exc: + raise ImportExecutionError(f"Dataset import backend failed: {exc}") from exc + finally: + try: + if not preserve: + for path, identity in ((stage, stage_id), (recovery, recovery_id)): + if path is not None and identity is not None: + try: + self._remove(path, identity) + except OSError as exc: + raise ImportExecutionError( + f"Dataset cleanup failed; preserved recovery evidence at {path}: {exc}" + ) from exc + finally: + for identity in (new_id, old_id, stage_id, recovery_id): + if identity is not None: + SplitService._close_stage_identity(identity) diff --git a/src/pt_snap_cli/core/dataset_resolver.py b/src/pt_snap_cli/core/dataset_resolver.py index d7b5296..a1d67ed 100644 --- a/src/pt_snap_cli/core/dataset_resolver.py +++ b/src/pt_snap_cli/core/dataset_resolver.py @@ -1,7 +1,8 @@ """Resolve finalized compatibility datasets above the single-DB Context layer. Original local reader of the P0 contract; no pickle, migration or producer code. -Native-v2 private replay staging has no published manifest and is not a dataset. +Native-v2 private replay staging is not a dataset until the native publisher closes, +attests and validates its complete versioned manifest. """ from __future__ import annotations @@ -16,10 +17,20 @@ DatasetValidation, DeviceRecord, QueryScope, + _unique_object, parse_manifest, validate_dataset, ) +from pt_snap_cli.core.dataset_files import hash_file as _hash_file +from pt_snap_cli.core.dataset_files import require_readonly_member as _require_readonly_member from pt_snap_cli.core.errors import DatabaseSchemaError, InvalidDeviceError, InvalidParameterError +from pt_snap_cli.core.native_dataset_contract import ( + NATIVE_FORMAT, + NativeManifest, + parse_native_manifest, + validate_native_dataset, + validate_native_scope, +) @dataclass(frozen=True) @@ -34,7 +45,7 @@ def device_ids(self) -> list[int]: @property def callstack_layout(self) -> str: - return "v1" + return "v2" if isinstance(self.validation.manifest, NativeManifest) else "v1" @property def callstack_layout_error(self) -> None: @@ -50,7 +61,11 @@ def device(self, device_id: int | None = None) -> DeviceRecord: def paths(self, scope: QueryScope) -> tuple[Path, ...]: """Address all matching slices; this does not execute/aggregate queries.""" try: - scope.validate(self.validation.manifest) + manifest = self.validation.manifest + if isinstance(manifest, NativeManifest): + validate_native_scope(scope, manifest) + else: + scope.validate(manifest) except DatasetContractError as exc: raise InvalidParameterError(str(exc)) from exc return tuple( @@ -69,7 +84,7 @@ def to_dict(self) -> dict[str, object]: "manifest_path": str(self.root / "manifest.json"), "schema_version": manifest.schema_version, "status": manifest.status, - "format": "compatibility-v1", + "format": NATIVE_FORMAT if isinstance(manifest, NativeManifest) else "compatibility-v1", "fingerprint": self.fingerprint, "real_event_count": self.validation.real_event_count, "boundary_event_count": self.validation.boundary_event_count, @@ -116,15 +131,23 @@ def inspect(self, path: Path | str) -> ResolvedDataset | None: raise DatabaseSchemaError("Dataset manifest must not be a symlink.") before = _hash_file(manifest_path) with manifest_path.open(encoding="utf-8") as source: - raw: object = json.load(source) - planned = parse_manifest(raw) + raw: object = json.load(source, object_pairs_hook=_unique_object) + planned = ( + parse_native_manifest(raw) + if isinstance(raw, dict) and raw.get("format") == NATIVE_FORMAT + else parse_manifest(raw) + ) # mode=ro can create WAL/SHM files even after the last writer has # checkpointed and removed them. Check every member before the first # SQLite open, without following aliases or repairing the artifact. for device in planned.devices: for item in device.slices: _require_readonly_member(root, root / item.file) - validation = validate_dataset(root) + validation = ( + validate_native_dataset(root, planned) + if isinstance(planned, NativeManifest) + else validate_dataset(root) + ) digest = hashlib.sha256() digest.update(before.encode("ascii")) for device in validation.manifest.devices: @@ -138,32 +161,3 @@ def inspect(self, path: Path | str) -> ResolvedDataset | None: return ResolvedDataset(root, validation, digest.hexdigest()) except (DatasetContractError, OSError, UnicodeError, json.JSONDecodeError) as exc: raise DatabaseSchemaError(f"Invalid dataset: {exc}") from exc - - -def _require_readonly_member(root: Path, member: Path) -> None: - # Manifest parsing fixes canonical relative paths; reject member/parent - # symlinks before the header read, not only when the validator opens SQLite. - parts = member.relative_to(root).parts - for index in range(1, len(parts) + 1): - if root.joinpath(*parts[:index]).is_symlink(): - raise DatabaseSchemaError(f"Dataset member must not use symlink paths: {member}") - if member.resolve() != member or not member.is_file(): - raise DatabaseSchemaError(f"Dataset member is missing or not a canonical file: {member}") - for suffix in ("-wal", "-journal", "-shm"): - sidecar = Path(str(member) + suffix) - if sidecar.exists() or sidecar.is_symlink(): - raise DatabaseSchemaError(f"Dataset member has a live SQLite sidecar: {member}") - with member.open("rb") as source: - header = source.read(20) - # SQLite's file-format write/read versions at offsets 18/19 mark WAL as 2. - # A sidecar-free WAL database is still unsafe to open with mode=ro. - if header[:16] == b"SQLite format 3\x00" and 2 in header[18:20]: - raise DatabaseSchemaError(f"Dataset member uses persistent WAL mode: {member}") - - -def _hash_file(path: Path) -> str: - digest = hashlib.sha256() - with path.open("rb") as source: - for chunk in iter(lambda: source.read(1024 * 1024), b""): - digest.update(chunk) - return digest.hexdigest() diff --git a/src/pt_snap_cli/core/focus_service.py b/src/pt_snap_cli/core/focus_service.py index 9e6015a..23e0b0b 100644 --- a/src/pt_snap_cli/core/focus_service.py +++ b/src/pt_snap_cli/core/focus_service.py @@ -1,6 +1,11 @@ from __future__ import annotations +import copy +import os import sqlite3 +import tempfile +from collections.abc import Iterator +from contextlib import contextmanager from pathlib import Path from typing import Any @@ -74,6 +79,121 @@ def get_focus( callstack_layout_error=callstack_layout_error, ) + @staticmethod + def _safe_focus_path(path: Path) -> None: + if path.resolve() != path or path.is_symlink() or (path.exists() and not path.is_file()): + raise OSError(f"Unsafe project focus path: {path}") + + @staticmethod + def _checkpoint_file(path: Path, content: bytes, mode: int) -> tuple[Path, tuple[int, int]]: + fd, name = tempfile.mkstemp(dir=path.parent, prefix=".focus.pt-snap-", suffix=".recovery") + backup = Path(name) + identity = os.fstat(fd) + try: + remaining = memoryview(content) + while remaining: + written = os.write(fd, remaining) + if written <= 0: + raise OSError("Could not write focus recovery checkpoint") + remaining = remaining[written:] + os.fchmod(fd, mode) + os.fsync(fd) + FocusService._verify_checkpoint(backup, (identity.st_dev, identity.st_ino)) + except BaseException: + try: + FocusService._verify_checkpoint(backup, (identity.st_dev, identity.st_ino)) + backup.unlink() + except OSError: + pass + raise + finally: + os.close(fd) + return backup, (identity.st_dev, identity.st_ino) + + @staticmethod + def _verify_checkpoint(path: Path, identity: tuple[int, int]) -> None: + FocusService._safe_focus_path(path) + info = path.lstat() + if (info.st_dev, info.st_ino) != identity: + raise OSError(f"Focus recovery checkpoint substituted: {path}") + + @staticmethod + def _restore_project_focus( + path: Path, backup: Path, identity: tuple[int, int], expected_bytes: bytes + ) -> None: + FocusService._safe_focus_path(path) + FocusService._verify_checkpoint(backup, identity) + if backup.read_bytes() != expected_bytes: + raise OSError("Focus recovery checkpoint content changed; preserve evidence") + os.replace(backup, path) + + @contextmanager + def project_focus_transaction(self) -> Iterator[None]: + """Import-only compensation, including writes that mutate and THEN raise. + + Store old bytes verbatim, not normalized JSON. An I/O/alias failure during + compensation preserves the checkpoint and reports its recovery location. + This is not a concurrency lock or an arbitrary-crash atomicity promise. + """ + from .errors import ImportExecutionError + + path = self._config.project_focus_path() + self._safe_focus_path(path) + old = path.read_bytes() if path.exists() else None + state = copy.deepcopy(self._config._config) + backup: Path | None = None + identity: tuple[int, int] | None = None + path.parent.mkdir(parents=True, exist_ok=True) + backup, identity = self._checkpoint_file( + path, + old if old is not None else b"", + path.stat().st_mode & 0o777 if old is not None else 0o600, + ) + preserve = False + try: + yield + except BaseException as focus_error: + self._config._config = state + try: + self._safe_focus_path(path) + if old is None: + if path.exists(): + # Verified exact path; never unlink a symlink target. + path.unlink() + elif not path.exists() or path.read_bytes() != old: + assert backup is not None and identity is not None + try: + self._restore_project_focus(path, backup, identity, old) + except BaseException: + # A rename may have restored the exact bytes BEFORE an + # injected/late I/O exception. Verify the actual outcome. + self._safe_focus_path(path) + if not path.exists() or path.read_bytes() != old: + raise + self._safe_focus_path(path) + if not path.exists() or path.read_bytes() != old: + raise OSError( + "Project focus was not restored exactly; preserve recovery evidence" + ) + if not backup.exists(): + backup = None + except BaseException as rollback_error: + preserve = True + raise ImportExecutionError( + f"Focus update failed and focus rollback also failed: {rollback_error}. " + f"Recovery focus checkpoint: {backup}; prior focus existed: {old is not None}." + ) from focus_error + raise + finally: + if backup is not None and identity is not None and not preserve: + # Failure to clean a committed checkpoint is safer than masking + # the real transaction outcome or deleting a substituted path. + try: + self._verify_checkpoint(backup, identity) + backup.unlink() + except OSError: + pass + def set_project_focus( self, db_path: Path | str, diff --git a/src/pt_snap_cli/core/import_metadata.py b/src/pt_snap_cli/core/import_metadata.py index 12c8747..07da314 100644 --- a/src/pt_snap_cli/core/import_metadata.py +++ b/src/pt_snap_cli/core/import_metadata.py @@ -1,11 +1,10 @@ from __future__ import annotations import hashlib -import re import sqlite3 from contextlib import closing from dataclasses import asdict -from datetime import datetime, timedelta, timezone +from datetime import datetime, timezone from pathlib import Path from typing import Any @@ -13,11 +12,16 @@ from pt_snap_cli.context import Context, DatabaseNotFoundError, SchemaVersionError from pt_snap_cli.core.dataset_resolver import DatasetResolver from pt_snap_cli.core.errors import DatabaseMissingError, DatabaseSchemaError, ImportMetadataError +from pt_snap_cli.core.import_metadata_contract import ( + metadata_from_mapping, + validate_import_metadata, +) from pt_snap_cli.core.models import ( CacheDecision, ImportMetadata, MetadataInspection, ) +from pt_snap_cli.core.native_dataset_contract import NativeManifest METADATA_TABLE = "pt_snap_metadata" METADATA_SCHEMA_VERSION = 1 @@ -27,7 +31,6 @@ IMPORTER_NAME = "pt-snap-cli" HASH_CHUNK_SIZE = 1024 * 1024 -_SHA256_RE = re.compile(r"^[0-9a-f]{64}$") _REQUIRED_COLUMNS = { "id", "metadata_schema_version", @@ -80,6 +83,8 @@ def inspect(self, db_path: Path | str) -> MetadataInspection: dataset = DatasetResolver().inspect(db_path) path = Path(db_path).expanduser().resolve() if dataset is not None: + if isinstance(dataset.validation.manifest, NativeManifest): + return MetadataInspection(path, "available", dataset.validation.manifest.metadata) # Per-slice extensions are not a dataset-wide importer/source attestation. return MetadataInspection(path, "unavailable", reason="metadata_missing") try: @@ -236,19 +241,7 @@ def inspection_to_dict(inspection: MetadataInspection) -> dict[str, Any]: } def _metadata_from_row(self, row: dict[str, Any]) -> ImportMetadata: - metadata = ImportMetadata( - metadata_schema_version=self._required_int(row["metadata_schema_version"]), - import_format_version=self._required_int(row["import_format_version"]), - source_sha256=self._required_str(row["source_sha256"]), - source_size=self._required_int(row["source_size"]), - source_name=self._required_str(row["source_name"]), - requested_device=self._optional_int(row["requested_device"]), - importer_name=self._required_str(row["importer_name"]), - importer_version=self._required_str(row["importer_version"]), - completed_at=self._required_str(row["completed_at"]), - ) - self._validate_metadata(metadata, allow_unsupported_version=True) - return metadata + return metadata_from_mapping(row) @staticmethod def _validate_metadata( @@ -261,19 +254,7 @@ def _validate_metadata( and metadata.metadata_schema_version != METADATA_SCHEMA_VERSION ): raise ValueError("Unsupported metadata schema version") - if metadata.metadata_schema_version < 1 or metadata.import_format_version < 1: - raise ValueError("Metadata versions must be positive integers") - if not _SHA256_RE.fullmatch(metadata.source_sha256): - raise ValueError("source_sha256 must be a lowercase SHA-256 digest") - if metadata.source_size < 0 or not metadata.source_name: - raise ValueError("Source metadata is invalid") - if metadata.requested_device is not None and metadata.requested_device < 0: - raise ValueError("requested_device must be non-negative") - if not metadata.importer_name or not metadata.importer_version: - raise ValueError("Importer metadata is invalid") - parsed_time = datetime.fromisoformat(metadata.completed_at) - if parsed_time.tzinfo is None or parsed_time.utcoffset() != timedelta(0): - raise ValueError("completed_at must use UTC") + validate_import_metadata(metadata) @staticmethod def _required_int(value: Any) -> int: diff --git a/src/pt_snap_cli/core/import_metadata_contract.py b/src/pt_snap_cli/core/import_metadata_contract.py new file mode 100644 index 0000000..b63cb73 --- /dev/null +++ b/src/pt_snap_cli/core/import_metadata_contract.py @@ -0,0 +1,55 @@ +"""Pure import-metadata value contract shared by standalone and dataset readers.""" + +from __future__ import annotations + +import re +from collections.abc import Mapping +from datetime import datetime, timedelta +from typing import cast + +from .models import ImportMetadata + + +def required_int(value: object) -> int: + if type(value) is not int: + raise TypeError("Expected integer metadata value") + return cast(int, value) + + +def required_str(value: object) -> str: + if not isinstance(value, str) or not value: + raise TypeError("Expected non-empty metadata string") + return value + + +def validate_import_metadata(metadata: ImportMetadata) -> None: + if metadata.metadata_schema_version < 1 or metadata.import_format_version < 1: + raise ValueError("Metadata versions must be positive integers") + if not re.fullmatch(r"[0-9a-f]{64}", metadata.source_sha256): + raise ValueError("source_sha256 must be a lowercase SHA-256 digest") + if metadata.source_size < 0 or not metadata.source_name: + raise ValueError("Source metadata is invalid") + if metadata.requested_device is not None and metadata.requested_device < 0: + raise ValueError("requested_device must be non-negative") + if not metadata.importer_name or not metadata.importer_version: + raise ValueError("Importer metadata is invalid") + parsed_time = datetime.fromisoformat(metadata.completed_at) + if parsed_time.tzinfo is None or parsed_time.utcoffset() != timedelta(0): + raise ValueError("completed_at must use UTC") + + +def metadata_from_mapping(row: Mapping[str, object]) -> ImportMetadata: + device = row["requested_device"] + metadata = ImportMetadata( + required_int(row["metadata_schema_version"]), + required_int(row["import_format_version"]), + required_str(row["source_sha256"]), + required_int(row["source_size"]), + required_str(row["source_name"]), + None if device is None else required_int(device), + required_str(row["importer_name"]), + required_str(row["importer_version"]), + required_str(row["completed_at"]), + ) + validate_import_metadata(metadata) + return metadata diff --git a/src/pt_snap_cli/core/import_service.py b/src/pt_snap_cli/core/import_service.py index 57c3600..5996794 100644 --- a/src/pt_snap_cli/core/import_service.py +++ b/src/pt_snap_cli/core/import_service.py @@ -1,20 +1,37 @@ from __future__ import annotations +import json import logging +from contextlib import nullcontext from pathlib import Path +from pt_snap_cli.core.dataset_import_backend import DatasetImportBackend +from pt_snap_cli.core.dataset_resolver import DatasetResolver, _require_readonly_member from pt_snap_cli.core.errors import ( DatabaseSchemaError, FocusFileInvalidError, ImportExecutionError, ImportMetadataError, InvalidDeviceError, + InvalidParameterError, SnapshotFileInvalidError, SourceChangedError, ) from pt_snap_cli.core.focus_service import FocusService from pt_snap_cli.core.import_metadata import ImportMetadataService -from pt_snap_cli.core.models import FocusState, ImportMetadata, ImportOptions, ImportResult +from pt_snap_cli.core.models import ( + CacheMissReason, + FocusState, + ImportMetadata, + ImportOptions, + ImportResult, +) +from pt_snap_cli.core.native_dataset_contract import ( + NATIVE_FORMAT, + build_native_manifest, + cache_identity, +) +from pt_snap_cli.core.sharded_replay_service import ShardedReplayResult from pt_snap_cli.core.snapshot_import_backend import SnapshotImportBackend logger = logging.getLogger(__name__) @@ -30,13 +47,18 @@ def __init__( focus_service: FocusService | None = None, backend: SnapshotImportBackend | None = None, metadata_service: ImportMetadataService | None = None, + dataset_backend: DatasetImportBackend | None = None, ) -> None: self._focus_service: FocusService | None = focus_service self._backend: SnapshotImportBackend = backend or SnapshotImportBackend() self._metadata_service: ImportMetadataService = metadata_service or ImportMetadataService() + self._dataset_backend = dataset_backend or DatasetImportBackend() def import_snapshot(self, options: ImportOptions) -> ImportResult: + self._validate_options(options) self._validate_snapshot_file(options.snapshot_file) + if options.events_per_slice is not None: + return self._import_dataset(options) output_dir = self._resolve_output_dir(options.snapshot_file, options.output_dir) db_path = self._backend.target_db_path(options.snapshot_file, output_dir) @@ -117,6 +139,142 @@ def set_focus_after_publish(published_db_path: Path) -> None: cache_miss_reason=decision.reason, ) + @staticmethod + def _validate_options(options: ImportOptions) -> None: + capacity = options.events_per_slice + if capacity is not None and (type(capacity) is not int or capacity <= 0): + raise InvalidParameterError("events_per_slice must be a positive integer.") + if options.device is not None and (type(options.device) is not int or options.device < 0): + raise InvalidParameterError("device must be a nonnegative integer.") + selected_format = ( + options.format + if options.format is not None + else (NATIVE_FORMAT if capacity is not None else "single-db") + ) + if selected_format in ("compatibility-v1", "msinsight"): + raise InvalidParameterError( + "Compatibility/msinsight export is not available; use pt-snap-native-v2. No compatible output is generated." + ) + if selected_format not in ("single-db", NATIVE_FORMAT): + raise InvalidParameterError(f"Unsupported import format: {selected_format}") + if (capacity is not None) != (selected_format == NATIVE_FORMAT): + raise InvalidParameterError( + "pt-snap-native-v2 requires --events-per-slice; single-db forbids it." + ) + + def _import_dataset(self, options: ImportOptions) -> ImportResult: + assert options.events_per_slice is not None + capacity = options.events_per_slice + output = ( + (options.output_dir if options.output_dir is not None else options.snapshot_file.parent) + .expanduser() + .absolute() + ) + target = self._dataset_backend.target_path(options.snapshot_file, output) + if target.resolve() != target or target.is_symlink(): + raise ImportExecutionError("Dataset output must be canonical and non-symlink.") + source_hash = self._hash_source(options.snapshot_file) + existing = self._dataset_backend.inspect_target(target) + reason: CacheMissReason = "database_missing" + if existing is not None: + manifest, expected = existing + reason = "forced" if options.force else "dataset_identity_changed" + try: + dataset = DatasetResolver().inspect(target) + except (DatabaseSchemaError, OSError): + dataset = None + if not options.force: + reason = "database_invalid" + if ( + not options.force + and dataset is not None + and manifest.identity == cache_identity(source_hash, options.device, capacity) + ): + self._recheck_source(options.snapshot_file, source_hash) + focus_state = self._set_focus_if_requested(target, options) + return ImportResult( + target, + options.device, + focus_state, + True, + manifest.metadata, + None, + target, + tuple(dataset.device_ids), + sum(len(d.slices) for d in manifest.devices), + NATIVE_FORMAT, + manifest.omitted_devices, + ) + if not options.force: + raise ImportExecutionError( + f"Existing dataset is not reusable ({reason}); use --force to replace this recognized native artifact. Existing target and focus preserved." + ) + else: + expected = None + completed: ImportMetadata | None = None + generated: ShardedReplayResult | None = None + + def finalize(result: ShardedReplayResult) -> None: + nonlocal completed, generated + metadata = self._metadata_service.build_metadata( + options.snapshot_file, source_hash, options.device + ) + for item in result.slices: + _require_readonly_member(result.directory, item.file) + self._metadata_service.write(item.file, metadata) + manifest = build_native_manifest(result, metadata, capacity) + with (result.directory / "manifest.json").open("x", encoding="utf-8") as file: + json.dump(manifest, file, indent=2) + validated = DatasetResolver().inspect(result.directory) + if validated is None: + raise ImportExecutionError( + "Staged dataset validation did not produce a complete artifact." + ) + completed, generated = metadata, result + + focus_state: FocusState | None = None + + def focus(path: Path) -> None: + nonlocal focus_state + focus_state = self._set_focus_if_requested(path, options) + + self._dataset_backend.dump_to_dataset( + options.snapshot_file, + target, + capacity=capacity, + device=options.device, + expected=expected, + finalize=finalize, + post_publish=focus if options.set_focus else None, + pre_publish=lambda: self._recheck_source(options.snapshot_file, source_hash), + ) + assert completed is not None and generated is not None + return ImportResult( + target, + options.device, + focus_state, + False, + completed, + reason, + target, + tuple(sorted({s.device for s in generated.slices})), + len(generated.slices), + NATIVE_FORMAT, + generated.omitted_devices, + ) + + def _hash_source(self, source: Path) -> str: + try: + return self._metadata_service.calculate_sha256(source) + except ImportMetadataError as exc: + raise ImportExecutionError(str(exc)) from exc + + def _recheck_source(self, source: Path, initial_hash: str) -> None: + if self._hash_source(source) != initial_hash: + raise SourceChangedError( + "Snapshot source changed while import was running; existing artifact preserved." + ) + def _set_focus_if_requested( self, db_path: Path, @@ -127,10 +285,16 @@ def _set_focus_if_requested( if self._focus_service is None: self._focus_service = FocusService() try: - return self._focus_service.set_project_focus( - db_path=db_path, - device_id=options.device, + transaction = ( + self._focus_service.project_focus_transaction() + if options.events_per_slice is not None + else nullcontext() ) + with transaction: + return self._focus_service.set_project_focus( + db_path=db_path, + device_id=options.device, + ) except (FocusFileInvalidError, DatabaseSchemaError, InvalidDeviceError, OSError) as exc: raise ImportExecutionError( f"Imported database cannot be registered as focus: {exc}" diff --git a/src/pt_snap_cli/core/models.py b/src/pt_snap_cli/core/models.py index b0ba150..620eaf3 100644 --- a/src/pt_snap_cli/core/models.py +++ b/src/pt_snap_cli/core/models.py @@ -17,6 +17,7 @@ "import_format_changed", "device_changed", "forced", + "dataset_identity_changed", ] MetadataStatus = Literal["available", "unavailable", "invalid"] SplitFormat = Literal["pickle", "json"] @@ -134,8 +135,12 @@ class ImportOptions: Attributes: snapshot_file: Path to the input `.pkl` or `.pickle` snapshot. - output_dir: Directory where the generated `.db` is written. + output_dir: Parent of the generated standalone DB or native dataset. Defaults to the snapshot file's parent directory. + events_per_slice: Positive real-event capacity per device shard; None + preserves standalone behavior. Negative boundaries do not count. + format: None selects single-db or pt-snap-native-v2 from capacity; + compatibility-v1/msinsight remain explicitly unavailable. device: Optional device id to focus on. When None, all available devices are imported. set_focus: When True (default), also write project focus so subsequent @@ -147,6 +152,8 @@ class ImportOptions: device: int | None = None set_focus: bool = True force: bool = False + events_per_slice: int | None = None + format: str | None = None @dataclass(frozen=True) @@ -182,7 +189,9 @@ class ImportResult: """Result of `pt-snap import`. Attributes: - db_path: Path to the generated SQLite database. + db_path: Path to the standalone SQLite database or native dataset directory. + dataset_path: Native dataset directory, or None for standalone imports. + devices/slice_count/omitted_devices: Complete native output inventory. device_id: Device id used during import, or None when not specified. focus_state: The FocusState written to project focus when options.set_focus is True, otherwise None. @@ -194,6 +203,11 @@ class ImportResult: reused: bool metadata: ImportMetadata cache_miss_reason: CacheMissReason | None + dataset_path: Path | None = None + devices: tuple[int, ...] = () + slice_count: int = 0 + format: str = "single-db" + omitted_devices: tuple[int, ...] = () @dataclass(frozen=True) diff --git a/src/pt_snap_cli/core/native_dataset_contract.py b/src/pt_snap_cli/core/native_dataset_contract.py new file mode 100644 index 0000000..729269f --- /dev/null +++ b/src/pt_snap_cli/core/native_dataset_contract.py @@ -0,0 +1,577 @@ +"""Original published pt-snap native-v2 protocol; NOT compatibility-v1/msinsight. + +Counts/capacity use original positions per device. IDs stay sparse and original. +Only closed, finalized artifacts are accepted; unknown extensions grant no capabilities. +""" + +from __future__ import annotations + +import json +import re +import sqlite3 +from contextlib import closing +from dataclasses import asdict, dataclass +from pathlib import Path +from typing import cast + +from .dataset_contract import ( + ACTION_NAMES, + BLOCK_COLUMNS, + BLOCK_STATES, + DICTIONARY_COLUMNS, + TRACE_COLUMNS, + DatasetContractError, + DatasetManifest, + DatasetValidation, + DeviceRecord, + QueryScope, + SliceRecord, + _array, + _integer, + _need, + _object, + _safe_file, + _schema, + _text, + _unique_object, +) +from .dataset_files import hash_file as _hash_file +from .dataset_files import require_readonly_member as _require_readonly_member +from .import_metadata_contract import metadata_from_mapping +from .models import ImportMetadata +from .sharded_replay_service import ShardedReplayResult + +NATIVE_FORMAT = "pt-snap-native-v2" +NATIVE_MANIFEST_VERSION = 1 +# Semantic contracts, deliberately independent of the installed CLI __version__. +NATIVE_FORMAT_VERSION = 1 +NATIVE_IMPORT_VERSION = 1 +NATIVE_EXTENSION_VERSION = 1 + + +@dataclass(frozen=True) +class NativeSlice(SliceRecord): + start_position: int + end_position: int + sha256: str + + +@dataclass(frozen=True) +class NativeManifest(DatasetManifest): + identity: dict[str, object] + metadata: ImportMetadata + omitted_devices: tuple[int, ...] + + +def cache_identity(source_sha256: str, device: int | None, capacity: int) -> dict[str, object]: + return { + "sourceSha256": source_sha256, + "requestedDevice": device, + "eventsPerSlice": capacity, + "outputFormat": NATIVE_FORMAT, + "manifestContractVersion": NATIVE_MANIFEST_VERSION, + "formatContractVersion": NATIVE_FORMAT_VERSION, + "importContractVersion": NATIVE_IMPORT_VERSION, + "extensionContractVersion": NATIVE_EXTENSION_VERSION, + "metadataSchemaVersion": 1, + "importFormatVersion": 2, + } + + +def _digest(value: object, loc: str) -> str: + result = _text(value, loc) + _need(re.fullmatch(r"[0-9a-f]{64}", result) is not None, loc, "hash", "expected SHA256") + return result + + +def parse_native_manifest(value: object) -> NativeManifest: + + data = _object(value, "manifest") + _need(data.get("format") == NATIVE_FORMAT, "format", "format", "unsupported native format") + version = _integer(data.get("schemaVersion"), "schemaVersion", minimum=1) + _need( + version == NATIVE_MANIFEST_VERSION, + "schemaVersion", + "version", + "unsupported native manifest", + ) + _need( + data.get("status") == "complete", "status", "building", "complete native dataset required" + ) + source = _text(data.get("sourceFile"), "sourceFile") + identity = _object(data.get("identity"), "identity") + _digest(identity.get("sourceSha256"), "identity.sourceSha256") + requested = identity.get("requestedDevice") + if requested is not None: + _integer(requested, "identity.requestedDevice", maximum=2**31 - 1) + capacity = _integer(identity.get("eventsPerSlice"), "identity.eventsPerSlice", minimum=1) + _need( + identity.get("outputFormat") == NATIVE_FORMAT, + "identity", + "format", + "format identity mismatch", + ) + for key in ( + "manifestContractVersion", + "formatContractVersion", + "importContractVersion", + "extensionContractVersion", + "metadataSchemaVersion", + "importFormatVersion", + ): + _integer(identity.get(key), "identity." + key, minimum=1) + _need( + set(identity) == set(cache_identity("", None, 1)), + "identity", + "fields", + "identity fields differ", + ) + try: + metadata = metadata_from_mapping(_object(data.get("metadata"), "metadata")) + except (KeyError, TypeError, ValueError) as exc: + raise DatasetContractError("metadata", "metadata", str(exc)) from exc + _need( + ( + metadata.source_sha256, + metadata.requested_device, + metadata.metadata_schema_version, + metadata.import_format_version, + ) + == ( + identity["sourceSha256"], + requested, + identity["metadataSchemaVersion"], + identity["importFormatVersion"], + ), + "metadata", + "identity", + "metadata disagrees with dataset identity", + ) + _need( + source == metadata.source_name, + "sourceFile", + "metadata", + "source name disagrees with metadata", + ) + omitted = tuple( + _integer(v, "omittedDevices", maximum=2**31 - 1) + for v in _array(data.get("omittedDevices"), "omittedDevices") + ) + _need(len(set(omitted)) == len(omitted), "omittedDevices", "duplicate", "duplicate device") + devices: list[DeviceRecord] = [] + for key, raw in _object(data.get("devices"), "devices").items(): + loc = f"devices.{key}" + _need( + isinstance(key, str) and re.fullmatch(r"0|[1-9][0-9]*", key) is not None, + loc, + "device", + "noncanonical device", + ) + device_id = _integer(int(key), loc, maximum=2**31 - 1) + device = _object(raw, loc) + count = _integer(device.get("eventCount"), loc + ".eventCount", minimum=1) + records = _array(device.get("slices"), loc + ".slices") + _need( + bool(records) + and device.get("sliceCount") == len(records) + and type(device.get("sliceCount")) is int, + loc, + "count", + "invalid sliceCount", + ) + ready = tuple( + _integer(v, loc + ".readySlices") + for v in _array(device.get("readySlices"), loc + ".readySlices") + ) + _need( + ready == tuple(range(len(records))), + loc, + "readiness", + "all slices must be ready in order", + ) + slices: list[SliceRecord] = [] + position, last_id = 0, -1 + for index, raw_slice in enumerate(records): + item = _object(raw_slice, loc) + _need( + type(item.get("index")) is int + and item["index"] == index + and item.get("ready") is True, + loc, + "index", + "slice index/readiness mismatch", + ) + start = _integer(item.get("startPosition"), loc + ".startPosition") + end = _integer(item.get("endPosition"), loc + ".endPosition", minimum=start) + _need( + start == position and end - start + 1 <= capacity, + loc, + "coverage", + "invalid positional coverage/capacity", + ) + # All but the last shard must be full; no omitted positions are allowed. + _need( + index == len(records) - 1 or end - start + 1 == capacity, + loc, + "capacity", + "nonfinal slice is not full", + ) + low = _integer(item.get("startEventId"), loc + ".startEventId", minimum=last_id + 1) + high = _integer(item.get("endEventId"), loc + ".endEventId", minimum=low) + file = _text(item.get("file"), loc + ".file") + _need( + file == f"device_{device_id}/slice_{index:05d}.db", + loc, + "path", + "noncanonical member", + ) + slices.append( + NativeSlice( + index, low, high, file, True, start, end, _digest(item.get("sha256"), loc) + ) + ) + position, last_id = end + 1, high + _need(position == count, loc, "coverage", "positions do not cover eventCount") + devices.append(DeviceRecord(device_id, count, ready, tuple(slices))) + _need(bool(devices), "devices", "empty", "event-bearing device required") + selected = {d.device_id for d in devices} + _need( + not selected.intersection(omitted), "omittedDevices", "device", "selected device is omitted" + ) + _need( + requested is None or selected == {requested}, + "identity", + "device", + "requested device mismatch", + ) + return NativeManifest( + version, "complete", source, "", capacity, tuple(devices), identity, metadata, omitted + ) + + +def read_native_manifest(root: Path) -> NativeManifest: + try: + with _safe_file(root, "manifest.json").open(encoding="utf-8") as file: + return parse_native_manifest(json.load(file, object_pairs_hook=_unique_object)) + except (OSError, UnicodeError, json.JSONDecodeError) as exc: + raise DatasetContractError("manifest.json", "read", str(exc)) from exc + + +def require_native_members(root: Path, manifest: NativeManifest) -> None: + """Ownership recognition before any replacement: never adopt unknown files/dirs.""" + _need( + root.resolve() == root and not root.is_symlink(), + "dataset", + "path", + "noncanonical directory", + ) + expected = {"manifest.json", *(f"device_{d.device_id}" for d in manifest.devices)} + _need( + {p.name for p in root.iterdir()} == expected, + "dataset", + "members", + "unexpected/missing dataset member", + ) + for device in manifest.devices: + directory = root / f"device_{device.device_id}" + _need( + not directory.is_symlink() and directory.is_dir(), + str(directory), + "path", + "unsafe device directory", + ) + _need( + {p.name for p in directory.iterdir()} == {Path(s.file).name for s in device.slices}, + str(directory), + "members", + "unexpected/missing shard", + ) + for item in device.slices: + _safe_file(root, item.file) + + +def validate_native_dataset(root: Path, manifest: NativeManifest) -> DatasetValidation: + + require_native_members(root, manifest) + expected_identity = cache_identity( + manifest.metadata.source_sha256, + manifest.metadata.requested_device, + manifest.events_per_slice, + ) + _need( + manifest.identity == expected_identity, + "identity", + "version", + "unsupported native semantic contract", + ) + # Check EVERY member before the FIRST SQLite open. No WAL/sidecars/repair. + for device in manifest.devices: + for raw in device.slices: + item = cast(NativeSlice, raw) + path = root / item.file + _require_readonly_member(root, path) + _need(_hash_file(path) == item.sha256, item.file, "hash", "member content changed") + boundaries = 0 + for device in manifest.devices: + ids: set[int] = set() + lifetimes: dict[int, tuple[object, ...]] = {} + references: list[tuple[int, int]] = [] + for raw in device.slices: + item = cast(NativeSlice, raw) + path = root / item.file + try: + with closing(sqlite3.connect(path.as_uri() + "?mode=ro", uri=True)) as conn: + conn.execute("PRAGMA query_only=ON") + _need( + conn.execute("PRAGMA quick_check").fetchall() == [("ok",)], + item.file, + "sqlite", + "integrity check failed", + ) + trace, block = f"trace_entry_{device.device_id}", f"block_{device.device_id}" + names = { + row[0] + for row in conn.execute( + "SELECT name FROM sqlite_master WHERE type IN ('table','view')" + ) + } + _need( + {n for n in names if n.lower().startswith(("trace_entry_", "block_"))} + == {trace, block}, + item.file, + "device_tables", + "wrong device/table set", + ) + for table, columns in ( + (trace, TRACE_COLUMNS[:-1] + (("callstackId", "INTEGER"),)), + (block, BLOCK_COLUMNS), + ("dictionary", DICTIONARY_COLUMNS), + ("callstack", (("id", "INTEGER"), ("callstack", "TEXT"))), + ): + _schema(conn, table, columns, item.file) + _need( + conn.execute( + "SELECT type FROM sqlite_master WHERE name='pt_snap_block_reference'" + ).fetchone() + == ("table",), + item.file, + "extension", + "finalized reference table required", + ) + reference_columns = conn.execute( + "PRAGMA table_info(pt_snap_block_reference)" + ).fetchall() + _need( + tuple((r[1], r[2].upper(), r[5]) for r in reference_columns) + == ( + ("blockId", "INTEGER", 1), + ("stream", "INTEGER", 0), + ("allocCallstack", "TEXT", 0), + ("freeCallstack", "TEXT", 0), + ), + item.file, + "extension", + "invalid reference schema", + ) + for table, column, values in ( + (trace, "action", tuple(enumerate((*ACTION_NAMES, "oom")))), + (block, "state", BLOCK_STATES), + ): + rows = conn.execute( + 'SELECT "key","value" FROM dictionary WHERE "table"=? AND "column"=?', + (table, column), + ).fetchall() + expected = {(str(k), v) for k, v in values} + _need( + len(rows) == len(expected) and set(rows) == expected, + item.file, + "dictionary", + "invalid native dictionary", + ) + real = [ + r[0] + for r in conn.execute(f'SELECT id FROM "{trace}" WHERE id>=0 ORDER BY id') + ] + _need( + len(real) == item.end_position - item.start_position + 1 + and (real[0], real[-1]) == (item.start_event_id, item.end_event_id) + and not ids.intersection(real), + item.file, + "event_identity", + "real IDs/count differ from positions/ranges", + ) + ids.update(real) + numeric = " OR ".join( + f"typeof({name})!='integer'" for name, _ in TRACE_COLUMNS[:-1] + ) + invalid = conn.execute( + f"SELECT id FROM \"{trace}\" WHERE {numeric} OR action NOT BETWEEN 0 AND 8 OR (id<0 AND action NOT IN (0,2)) OR size<0 OR allocated<0 OR active<0 OR reserved<0 OR typeof(callstackId)!='integer' OR callstackId NOT IN (SELECT id FROM callstack) LIMIT 1" + ).fetchone() + _need( + invalid is None, + item.file, + "event_values", + "invalid native event/scalar/callstack", + ) + _need( + conn.execute( + "SELECT id FROM callstack WHERE typeof(callstack) NOT IN ('text','null') LIMIT 1" + ).fetchone() + is None, + item.file, + "callstack", + "invalid text stack", + ) + boundaries += conn.execute( + f'SELECT COUNT(*) FROM "{trace}" WHERE id<0' + ).fetchone()[0] + stacks = { + r[0]: r[1:] for r in conn.execute("SELECT * FROM pt_snap_block_reference") + } + blocks = conn.execute(f'SELECT * FROM "{block}"').fetchall() + _need( + set(stacks) == {r[0] for r in blocks}, + item.file, + "extension", + "reference/block membership differs", + ) + for row in blocks: + bid, address, size, requested, state, alloc, free = row + _need( + all(type(v) is int for v in row) + and state in (-1, 0, 1) + and size >= 0 + and 0 <= requested <= size + and ((alloc == -1 and bid < 0) or (alloc >= 0 and bid == alloc)) + and (free == -1 or free >= 0 and (alloc == -1 or free >= alloc)), + item.file, + "block_identity", + "unfinalized/invalid block", + ) + ref = stacks[bid] + _need( + type(ref[0]) is int + and all(v is None or isinstance(v, str) for v in ref[1:]), + item.file, + "extension", + "invalid reference values", + ) + signature = (address, size, requested, alloc, free, *ref) + _need( + bid not in lifetimes or lifetimes[bid] == signature, + item.file, + "block_identity", + "conflicting finalized identity/reference", + ) + lifetimes[bid] = signature + references.append((alloc, free)) + except sqlite3.DatabaseError as exc: + raise DatasetContractError(item.file, "sqlite", str(exc)) from exc + try: + with closing( + sqlite3.connect(path.as_uri() + "?mode=ro", uri=True) + ) as metadata_conn: + metadata_conn.row_factory = sqlite3.Row + _need( + metadata_conn.execute( + "SELECT type FROM sqlite_master WHERE name='pt_snap_metadata'" + ).fetchone()[0] + == "table", + item.file, + "metadata", + "metadata must be an actual table", + ) + rows = metadata_conn.execute("SELECT * FROM pt_snap_metadata").fetchall() + _need( + len(rows) == 1 + and rows[0]["id"] == 1 + and set(rows[0].keys()) == {"id", *asdict(manifest.metadata)}, + item.file, + "metadata", + "invalid metadata row/schema", + ) + metadata = metadata_from_mapping(dict(rows[0])) + _need( + metadata == manifest.metadata, + item.file, + "metadata", + "member metadata differs from manifest", + ) + except (sqlite3.DatabaseError, KeyError, TypeError, ValueError) as exc: + raise DatasetContractError(item.file, "metadata", str(exc)) from exc + _require_readonly_member(root, path) + _need( + _hash_file(path) == item.sha256, + item.file, + "hash", + "member changed during validation", + ) + _need( + len(ids) == device.event_count + and all(v == -1 or v in ids for pair in references for v in pair), + "devices", + "event_identity", + "count/lifetime references not in original device scope", + ) + return DatasetValidation(manifest, sum(d.event_count for d in manifest.devices), boundaries) + + +def validate_native_scope(scope: QueryScope, manifest: NativeManifest) -> None: + if scope.kind == "event_range": + device = next((d for d in manifest.devices if d.device_id == scope.device_id), None) + _need( + type(scope.device_id) is int and device is not None, "scope", "device", "unknown device" + ) + assert device is not None + _need(scope.slice_index is None, "scope", "selector", "unexpected slice selector") + start = _integer( + scope.start_event_id, + "scope.start_event_id", + minimum=device.slices[0].start_event_id, + maximum=device.slices[-1].end_event_id, + ) + _integer( + scope.end_event_id, + "scope.end_event_id", + minimum=start, + maximum=device.slices[-1].end_event_id, + ) + else: + scope.validate(manifest) + + +def build_native_manifest( + result: ShardedReplayResult, metadata: ImportMetadata, capacity: int +) -> dict[str, object]: + + devices: dict[str, object] = {} + for device in sorted({item.device for item in result.slices}): + slices = [s for s in result.slices if s.device == device] + devices[str(device)] = { + "eventCount": sum(s.end_position - s.start_position + 1 for s in slices), + "sliceCount": len(slices), + "readySlices": list(range(len(slices))), + "slices": [ + { + "index": s.index, + "startPosition": s.start_position, + "endPosition": s.end_position, + "startEventId": s.start_event_id, + "endEventId": s.end_event_id, + "file": s.file.relative_to(result.directory).as_posix(), + "ready": True, + "sha256": _hash_file(s.file), + } + for s in slices + ], + } + return { + "format": NATIVE_FORMAT, + "schemaVersion": NATIVE_MANIFEST_VERSION, + "status": "complete", + "sourceFile": metadata.source_name, + "identity": cache_identity(metadata.source_sha256, metadata.requested_device, capacity), + "metadata": asdict(metadata), + "omittedDevices": list(result.omitted_devices), + "devices": devices, + } diff --git a/src/pt_snap_cli/core/overview_service.py b/src/pt_snap_cli/core/overview_service.py index a109c88..c8be173 100644 --- a/src/pt_snap_cli/core/overview_service.py +++ b/src/pt_snap_cli/core/overview_service.py @@ -14,7 +14,7 @@ ) from pt_snap_cli.core.focus_service import FocusService from pt_snap_cli.core.import_metadata import ImportMetadataService -from pt_snap_cli.core.models import DatabaseOverview, DeviceTraceBounds, MetadataInspection +from pt_snap_cli.core.models import DatabaseOverview, DeviceTraceBounds class OverviewService: @@ -56,12 +56,16 @@ def inspect( db_path=db_path, focus_source=resolved.source, devices=[ - DeviceTraceBounds(device.device_id, 0, device.event_count - 1) + DeviceTraceBounds( + device.device_id, + device.slices[0].start_event_id, + device.slices[-1].end_event_id, + ) for device in sorted( dataset.validation.manifest.devices, key=lambda d: d.device_id ) ], - metadata=MetadataInspection(db_path, "unavailable", reason="metadata_missing"), + metadata=self._metadata_service.inspect(db_path), dataset=dataset.to_dict(), ) ctx = self._validated_context(db_path) diff --git a/src/pt_snap_cli/core/query_service.py b/src/pt_snap_cli/core/query_service.py index d5013f3..2f8f2f5 100644 --- a/src/pt_snap_cli/core/query_service.py +++ b/src/pt_snap_cli/core/query_service.py @@ -315,7 +315,7 @@ def _dataset_query( item = device.slices[slice_index] low, high = item.start_event_id, item.end_event_id else: - low, high = 0, device.event_count - 1 + low, high = device.slices[0].start_event_id, device.slices[-1].end_event_id event_id = validated.get("id") start = validated.get("min_id") end = validated.get("max_id") diff --git a/tests/core/test_dataset_import.py b/tests/core/test_dataset_import.py new file mode 100644 index 0000000..54c2970 --- /dev/null +++ b/tests/core/test_dataset_import.py @@ -0,0 +1,429 @@ +"""Original deterministic temporary snapshots; no new committed pickle fixtures.""" + +import copy +import dataclasses +import hashlib +import json +import sqlite3 +from contextlib import closing +from pathlib import Path + +import pytest + +from pt_snap_cli.core import import_metadata, import_service, native_dataset_contract +from pt_snap_cli.core.dataset_import_backend import DatasetImportBackend +from pt_snap_cli.core.dataset_resolver import DatasetResolver +from pt_snap_cli.core.errors import ImportExecutionError, InvalidParameterError, SourceChangedError +from pt_snap_cli.core.focus_service import FocusService +from pt_snap_cli.core.import_service import ImportService +from pt_snap_cli.core.models import ImportOptions +from pt_snap_cli.core.native_dataset_contract import NATIVE_FORMAT +from pt_snap_cli.snapshot.tools.adaptors import sharded_replay +from tests.snapshot.test_sharded_replay import lifecycle_data, write_source + + +@pytest.fixture(autouse=True) +def isolate(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + monkeypatch.setenv("HOME", str(tmp_path)) + monkeypatch.setattr(Path, "home", lambda: tmp_path) + monkeypatch.delenv("PT_SNAP_DB_PATH", raising=False) + + +def source_options(tmp_path, *, multiple=False): + data = lifecycle_data() + for index, event in enumerate(data["device_traces"][0]): + event["id"] = index * 10 + 7 + if multiple: + data["device_traces"].append(copy.deepcopy(data["device_traces"][0])) + data["segments"] += [{**copy.deepcopy(s), "device": 1} for s in data["segments"]] + data["device_traces"].append([]) + return ImportOptions(write_source(tmp_path, data), tmp_path / "output", events_per_slice=2) + + +def hashes(root): + return { + str(p.relative_to(root)): hashlib.sha256(p.read_bytes()).hexdigest() + for p in root.rglob("*") + if p.is_file() + } + + +def saved_state(tmp_path): + service = ImportService() + options = source_options(tmp_path) + first = service.import_snapshot(options) + focus = tmp_path / ".pt-snap/focus.json" + return ( + service, + dataclasses.replace(options, force=True), + first, + hashes(first.db_path), + focus.read_bytes(), + ) + + +def assert_preserved(tmp_path, first, old, focus): + assert hashes(first.db_path) == old + assert (tmp_path / ".pt-snap/focus.json").read_bytes() == focus + assert list(first.db_path.parent.iterdir()) == [first.db_path] + + +def test_generation_complete_focus_result_and_all_devices(tmp_path): + options = source_options(tmp_path, multiple=True) + result = ImportService().import_snapshot(options) + assert result.dataset_path == result.db_path + assert result.devices == (0, 1) and result.omitted_devices == (2,) + assert result.slice_count == 10 and result.format == NATIVE_FORMAT + assert result.focus_state.callstack_layout == "v2" + dataset = DatasetResolver().inspect(result.db_path) + assert dataset.validation.real_event_count == 18 + assert dataset.validation.manifest.devices[0].event_count == 9 + assert ( + result.metadata.source_sha256 + == hashlib.sha256(options.snapshot_file.read_bytes()).hexdigest() + ) + manifest = json.loads((result.db_path / "manifest.json").read_text()) + assert "cacheHash" not in manifest and manifest["format"] == NATIVE_FORMAT + assert manifest["devices"]["0"]["slices"][-1]["endEventId"] == 87 + + +def test_cache_reuses_full_artifact_without_loading_or_cli_version_invalidation( + tmp_path, monkeypatch +): + options = dataclasses.replace(source_options(tmp_path), set_focus=False) + service = ImportService() + first = service.import_snapshot(options) + monkeypatch.setattr( + service._dataset_backend.replay, + "stage", + lambda *a, **k: pytest.fail("cache hit loaded source"), + ) + monkeypatch.setattr(import_metadata, "__version__", "future-cli-release") + second = service.import_snapshot(options) + assert second.reused and second.metadata == first.metadata + assert second.focus_state is None and not (tmp_path / ".pt-snap").exists() + + +@pytest.mark.parametrize( + "axis", + [ + "content", + "device", + "capacity", + "format", + "manifest", + "import", + "extension", + "metadata", + "db-layout", + ], +) +def test_cache_axes_require_force_and_rebuild(tmp_path, monkeypatch, axis): + service, options, first, old, focus = saved_state(tmp_path) + requested = dataclasses.replace(options, force=False) + if axis == "content": + options.snapshot_file.write_bytes(options.snapshot_file.read_bytes() + b"trailing-content") + elif axis == "device": + requested = dataclasses.replace(requested, device=0) + elif axis == "capacity": + requested = dataclasses.replace(requested, events_per_slice=3) + else: + constants = { + "format": (native_dataset_contract, "NATIVE_FORMAT_VERSION"), + "manifest": (native_dataset_contract, "NATIVE_MANIFEST_VERSION"), + "import": (native_dataset_contract, "NATIVE_IMPORT_VERSION"), + "extension": (native_dataset_contract, "NATIVE_EXTENSION_VERSION"), + "metadata": (import_metadata, "METADATA_SCHEMA_VERSION"), + "db-layout": (import_metadata, "IMPORT_FORMAT_VERSION"), + } + module, key = constants[axis] + # A manifest protocol upgrade cannot adopt/replace an unknown older schema. + # This axis is therefore exercised as identity invalidation, not schema relabeling. + if axis in ("manifest", "metadata", "db-layout"): + path = first.db_path / "manifest.json" + data = json.loads(path.read_text()) + identity_key = { + "manifest": "manifestContractVersion", + "metadata": "metadataSchemaVersion", + "db-layout": "importFormatVersion", + }[axis] + data["identity"][identity_key] += 1 + if axis != "manifest": + column = ( + "metadata_schema_version" if axis == "metadata" else "import_format_version" + ) + data["metadata"][column] += 1 + for device_data in data["devices"].values(): + for shard in device_data["slices"]: + member = first.db_path / shard["file"] + with closing(sqlite3.connect(member)) as conn, conn: + conn.execute( + f"UPDATE pt_snap_metadata SET {column}=?", + (data["metadata"][column],), + ) + shard["sha256"] = hashlib.sha256(member.read_bytes()).hexdigest() + path.write_text(json.dumps(data)) + old = hashes(first.db_path) + else: + monkeypatch.setattr(module, key, getattr(module, key) + 1) + with pytest.raises(ImportExecutionError, match="not reusable"): + service.import_snapshot(requested) + assert_preserved(tmp_path, first, old, focus) + result = service.import_snapshot(dataclasses.replace(requested, force=True)) + assert not result.reused and result.cache_miss_reason == "forced" + assert DatasetResolver().inspect(result.db_path) is not None + + +def test_corrupt_late_member_never_reuses_first_ready_shard(tmp_path): + service, options, first, _, focus = saved_state(tmp_path) + last = first.db_path / "device_0/slice_00004.db" + last.write_bytes(b"corrupt last member") + old = hashes(first.db_path) + with pytest.raises(ImportExecutionError, match="not reusable"): + service.import_snapshot(dataclasses.replace(options, force=False)) + assert_preserved(tmp_path, first, old, focus) + assert not service.import_snapshot(options).reused + + +@pytest.mark.parametrize( + "phase", + ["load", "write", "finalize", "registry-validation", "metadata", "manifest", "validation"], +) +def test_prepublication_failure_preserves_artifact_focus_and_closes_connections( + tmp_path, monkeypatch, phase +): + service, options, first, old, focus = saved_state(tmp_path) + handles = [] + connect = sqlite3.connect + + def counted(*args, **kwargs): + conn = connect(*args, **kwargs) + handles.append(conn) + return conn + + monkeypatch.setattr(sqlite3, "connect", counted) + + def fail(*args, **kwargs): + raise OSError(phase + " injected failure") + + if phase == "load": + monkeypatch.setattr( + "pt_snap_cli.core.sharded_replay_service.load_snapshot_representation", fail + ) + elif phase == "write": + monkeypatch.setattr(sharded_replay.SnapshotDbHandler, "insert_event", fail) + elif phase == "finalize": + monkeypatch.setattr(sharded_replay.BlockRegistry, "finalize", fail) + elif phase == "registry-validation": + monkeypatch.setattr(sharded_replay.BlockRegistry, "validate", fail) + elif phase == "metadata": + monkeypatch.setattr(service._metadata_service, "write", fail) + elif phase == "manifest": + monkeypatch.setattr(import_service.json, "dump", fail) + elif phase == "validation": + monkeypatch.setattr(import_service.DatasetResolver, "inspect", fail) + with pytest.raises(ImportExecutionError, match="injected failure"): + service.import_snapshot(options) + assert_preserved(tmp_path, first, old, focus) + for conn in handles: + with pytest.raises(sqlite3.ProgrammingError): + conn.execute("SELECT 1") + + +@pytest.mark.parametrize("move", ["old", "new"]) +@pytest.mark.parametrize("after_move", [False, True]) +def test_publication_rename_failure_even_after_mutation_compensates( + tmp_path, monkeypatch, move, after_move +): + service, options, first, old, focus = saved_state(tmp_path) + real = service._dataset_backend._move + fired = False + + def fault(source, dest, identity): + nonlocal fired + selected = ( + source == first.db_path + if move == "old" + else dest == first.db_path and source.name == "artifact" + ) + if selected and not fired: + fired = True + if after_move: + real(source, dest, identity) + raise OSError("publication injected failure") + real(source, dest, identity) + + monkeypatch.setattr(service._dataset_backend, "_move", fault) + with pytest.raises(ImportExecutionError, match="publication injected failure"): + service.import_snapshot(options) + assert_preserved(tmp_path, first, old, focus) + + +@pytest.mark.parametrize("existing", [False, True]) +def test_focus_mutates_before_raising_restores_exact_bytes_memory_and_artifact( + tmp_path, monkeypatch, existing +): + if existing: + service, options, first, old, focus = saved_state(tmp_path) + else: + service = ImportService() + options = source_options(tmp_path) + fs = FocusService() + fs._config._config = {"remember": ["old-memory-state"]} + service._focus_service = fs + + def mutate_then_fail(*a, **k): + path = tmp_path / ".pt-snap/focus.json" + path.write_bytes(b"mutated before raising") + fs._config._config["remember"].append("changed") + raise OSError("focus injected failure") + + monkeypatch.setattr(fs._config, "write_project_focus", mutate_then_fail) + with pytest.raises(ImportExecutionError, match="focus injected failure"): + service.import_snapshot(options) + assert fs._config._config == {"remember": ["old-memory-state"]} + if existing: + assert_preserved(tmp_path, first, old, focus) + else: + assert not (tmp_path / ".pt-snap/focus.json").exists() + assert not list(options.output_dir.iterdir()) + + +def test_source_change_after_validation_rejects_publication(tmp_path, monkeypatch): + service, options, first, old, focus = saved_state(tmp_path) + initial = service._metadata_service.calculate_sha256(options.snapshot_file) + sequence = iter([initial, "f" * 64]) + monkeypatch.setattr(service._metadata_service, "calculate_sha256", lambda p: next(sequence)) + with pytest.raises(SourceChangedError, match="source changed"): + service.import_snapshot(options) + assert_preserved(tmp_path, first, old, focus) + + +def test_directory_rollback_failure_keeps_recovery_bytes_and_reports(tmp_path, monkeypatch): + service, options, first, old, focus = saved_state(tmp_path) + real = service._dataset_backend._move + + def fault(source, dest, identity): + if source.name == "previous": + raise OSError("rollback injected failure") + real(source, dest, identity) + + monkeypatch.setattr(service._dataset_backend, "_move", fault) + monkeypatch.setattr( + service._focus_service._config, + "write_project_focus", + lambda *a, **k: (_ for _ in ()).throw(OSError("focus failed")), + ) + with pytest.raises(ImportExecutionError, match="rollback also failed.*Recovery directory"): + service.import_snapshot(options) + backup = list(options.output_dir.glob("*.recovery-*/previous")) + assert len(backup) == 1 and hashes(backup[0]) == old + assert (tmp_path / ".pt-snap/focus.json").read_bytes() == focus + + +def test_focus_rollback_failure_preserves_checkpoint_and_restores_old_dataset( + tmp_path, monkeypatch +): + service, options, first, old, focus = saved_state(tmp_path) + fs = service._focus_service + + def mutate(*a, **k): + (tmp_path / ".pt-snap/focus.json").write_bytes(b"changed") + raise OSError("focus failed") + + monkeypatch.setattr(fs._config, "write_project_focus", mutate) + monkeypatch.setattr( + fs, "_restore_project_focus", lambda *a: (_ for _ in ()).throw(OSError("restore failed")) + ) + with pytest.raises( + ImportExecutionError, match="focus rollback also failed.*Recovery focus checkpoint" + ): + service.import_snapshot(options) + backups = list((tmp_path / ".pt-snap").glob("*.recovery")) + assert len(backups) == 1 and backups[0].read_bytes() == focus + assert hashes(first.db_path) == old + + +@pytest.mark.parametrize("kind", ["empty", "unknown", "symlink", "dangling", "extra"]) +def test_force_never_deletes_unrecognized_targets_or_aliases(tmp_path, kind): + service = ImportService() + options = source_options(tmp_path) + target = DatasetImportBackend.target_path(options.snapshot_file, options.output_dir) + target.parent.mkdir() + if kind in ("symlink", "dangling"): + victim = tmp_path / "victim" + if kind == "symlink": + victim.mkdir() + (victim / "owner").write_text("safe") + target.symlink_to(victim, target_is_directory=True) + elif kind == "extra": + service.import_snapshot(options) + (target / "unrelated").write_text("preserve") + else: + target.mkdir() + if kind == "unknown": + (target / "owner").write_text("safe") + with pytest.raises(ImportExecutionError): + service.import_snapshot(dataclasses.replace(options, force=True)) + assert target.exists() or target.is_symlink() + if kind == "symlink": + assert (victim / "owner").read_text() == "safe" + if kind == "extra": + assert (target / "unrelated").read_text() == "preserve" + + +def test_concurrent_creation_is_not_replaced(tmp_path, monkeypatch): + service = ImportService() + options = dataclasses.replace(source_options(tmp_path), set_focus=False) + target = DatasetImportBackend.target_path(options.snapshot_file, options.output_dir) + real = service._dataset_backend._move + + def race(source, dest, identity): + if dest == target: + dest.mkdir() + (dest / "racing-owner").write_text("safe") + real(source, dest, identity) + + monkeypatch.setattr(service._dataset_backend, "_move", race) + with pytest.raises(ImportExecutionError): + service.import_snapshot(options) + assert (target / "racing-owner").read_text() == "safe" + assert list(target.parent.iterdir()) == [target] + + +@pytest.mark.parametrize( + "capacity,fmt,device", + [ + (0, None, None), + (True, None, None), + (2, "single-db", None), + (None, NATIVE_FORMAT, None), + (2, "compatibility-v1", None), + (2, "msinsight", None), + (2, "unknown", None), + (2, None, -1), + (2, None, True), + ], +) +def test_invalid_options_precede_loading(tmp_path, capacity, fmt, device): + with pytest.raises(InvalidParameterError): + ImportService().import_snapshot( + ImportOptions( + tmp_path / "missing.pkl", events_per_slice=capacity, format=fmt, device=device + ) + ) + + +def test_default_single_db_unchanged_and_separate_target(tmp_path): + options = source_options(tmp_path) + service = ImportService() + single = service.import_snapshot( + dataclasses.replace(options, events_per_slice=None, set_focus=False) + ) + native = service.import_snapshot(dataclasses.replace(options, set_focus=False)) + assert single.db_path.is_file() and single.dataset_path is None + assert single.format == "single-db" and native.db_path.is_dir() + assert service.import_snapshot( + dataclasses.replace(options, events_per_slice=None, set_focus=False) + ).reused diff --git a/tests/core/test_dataset_import_failures.py b/tests/core/test_dataset_import_failures.py new file mode 100644 index 0000000..852c895 --- /dev/null +++ b/tests/core/test_dataset_import_failures.py @@ -0,0 +1,234 @@ +import dataclasses +import os +from pathlib import Path + +import pytest + +from pt_snap_cli.core import sharded_replay_service +from pt_snap_cli.core.errors import ImportExecutionError, SourceChangedError +from pt_snap_cli.core.import_service import ImportService +from pt_snap_cli.core.split_service import SplitService +from tests.core.test_dataset_import import assert_preserved, hashes, saved_state, source_options + + +@pytest.fixture(autouse=True) +def isolate(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + monkeypatch.setenv("HOME", str(tmp_path)) + monkeypatch.setattr(Path, "home", lambda: tmp_path) + monkeypatch.delenv("PT_SNAP_DB_PATH", raising=False) + + +@pytest.mark.parametrize("fail", [False, True]) +def test_directory_descriptors_closed_on_success_and_failure(tmp_path, monkeypatch, fail): + service, options, first, old, focus = saved_state(tmp_path) + descriptors = [] + original = SplitService._stage_identity + + def track(path): + result = original(path) + if result[2] is not None: + descriptors.append(result[2]) + return result + + monkeypatch.setattr(SplitService, "_stage_identity", track) + if fail: + monkeypatch.setattr( + service._dataset_backend.replay, + "stage", + lambda *a, **k: (_ for _ in ()).throw(OSError("load failure")), + ) + with pytest.raises(ImportExecutionError): + service.import_snapshot(options) + assert_preserved(tmp_path, first, old, focus) + else: + service.import_snapshot(options) + assert descriptors + for fd in descriptors: + with pytest.raises(OSError): + os.fstat(fd) + + +def test_actual_source_mutation_during_load_rejects_before_publish(tmp_path, monkeypatch): + service, options, first, old, focus = saved_state(tmp_path) + original = sharded_replay_service.load_snapshot_representation + + def changed(path): + data = original(path) + path.write_bytes(path.read_bytes() + b"actual concurrent source change") + return data + + monkeypatch.setattr(sharded_replay_service, "load_snapshot_representation", changed) + with pytest.raises(SourceChangedError): + service.import_snapshot(options) + assert_preserved(tmp_path, first, old, focus) + + +def test_existing_target_changed_during_staging_is_not_adopted(tmp_path, monkeypatch): + service, options, first, old, focus = saved_state(tmp_path) + original = service._dataset_backend.replay.stage + + def changed(*a, **k): + result = original(*a, **k) + (first.db_path / "racing-owner").write_text("preserve") + return result + + monkeypatch.setattr(service._dataset_backend.replay, "stage", changed) + with pytest.raises(ImportExecutionError, match="preserved"): + service.import_snapshot(options) + assert (first.db_path / "racing-owner").read_text() == "preserve" + current = hashes(first.db_path) + current.pop("racing-owner") + assert current == old + assert (tmp_path / ".pt-snap/focus.json").read_bytes() == focus + assert list(options.output_dir.iterdir()) == [first.db_path] + + +def test_substituted_stage_is_not_deleted_or_followed(tmp_path, monkeypatch): + options = dataclasses.replace(source_options(tmp_path), set_focus=False) + service = ImportService() + original = service._dataset_backend.replay.stage + stages = [] + + def substituted(*a, **k): + result = original(*a, **k) + stage = result.directory.parent + stage.rename(tmp_path / "displaced-owned-stage") + stage.mkdir() + (stage / "unknown-owner").write_text("preserve") + stages.append(stage) + return result + + monkeypatch.setattr(service._dataset_backend.replay, "stage", substituted) + with pytest.raises(ImportExecutionError, match="cleanup failed.*preserved"): + service.import_snapshot(options) + assert (stages[0] / "unknown-owner").read_text() == "preserve" + assert not list(options.output_dir.glob("*.pt-snap-native-v2")) + + +def test_committed_recovery_cleanup_failure_preserves_old_bytes_reports_truthfully( + tmp_path, monkeypatch +): + service, options, first, old, _ = saved_state(tmp_path) + original = service._dataset_backend._remove + + def failed(path, identity): + if path.name == "previous": + raise OSError("cleanup injected failure") + original(path, identity) + + monkeypatch.setattr(service._dataset_backend, "_remove", failed) + with pytest.raises(ImportExecutionError, match="publication committed.*preserved evidence"): + service.import_snapshot(options) + backups = list(options.output_dir.glob("*.recovery-*/previous")) + assert len(backups) == 1 and hashes(backups[0]) == old + assert first.db_path.exists() # Do not falsely report rollback after commit. + + +@pytest.mark.parametrize("atomic_fault", [False, True]) +def test_real_focus_write_mutation_before_error_and_late_restore_error( + tmp_path, monkeypatch, atomic_fault +): + service, options, first, old, focus = saved_state(tmp_path) + options = dataclasses.replace(options, device=0) + path = tmp_path / ".pt-snap/focus.json" + if atomic_fault: + original = os.replace + + def fail_after_rename(source, destination): + original(source, destination) + if Path(destination) == path: + raise OSError("focus rename completed then failed") + + monkeypatch.setattr(os, "replace", fail_after_rename) + else: + original = service._focus_service._config.write_project_focus + + def fail_after_write(*a, **k): + original(*a, **k) + raise OSError("focus write completed then failed") + + monkeypatch.setattr(service._focus_service._config, "write_project_focus", fail_after_write) + with pytest.raises(ImportExecutionError, match="focus .*completed then failed") as error: + service.import_snapshot(options) + assert "rollback also failed" not in str(error.value) + assert_preserved(tmp_path, first, old, focus) + + +def test_parent_finding_substituted_recovery_is_preserved_and_reported(tmp_path, monkeypatch): + service, options, first, old, focus = saved_state(tmp_path) + original = service._dataset_backend._move + displaced = tmp_path / "safe-displaced-previous" + markers = [] + + def fault(source, dest, identity): + if source.name == "artifact": + previous = next(options.output_dir.glob("*.recovery-*/previous")) + assert previous.resolve() == previous and not previous.is_symlink() + assert displaced.resolve() == displaced and not displaced.exists() + previous.rename(displaced) + previous.mkdir() + marker = previous / "foreign-owner" + marker.write_text("must survive") + markers.append(marker) + raise OSError("publication failed after recovery substitution") + original(source, dest, identity) + + monkeypatch.setattr(service._dataset_backend, "_move", fault) + with pytest.raises( + ImportExecutionError, match="directory rollback also failed.*Recovery directory" + ): + service.import_snapshot(options) + assert hashes(displaced) == old + assert markers[0].read_text() == "must survive" + assert (tmp_path / ".pt-snap/focus.json").read_bytes() == focus + assert list(options.output_dir.glob("*.stage-*")) + + +def test_parent_finding_same_inode_focus_checkpoint_is_not_trusted(tmp_path, monkeypatch): + service, options, first, old, _ = saved_state(tmp_path) + original = service._focus_service._config.write_project_focus + checkpoints = [] + + def fault(*a, **k): + original(*a, **k) + backup = next((tmp_path / ".pt-snap").glob("*.recovery")) + inode = backup.stat().st_ino + backup.write_bytes(b"same inode, foreign checkpoint bytes") + assert backup.stat().st_ino == inode + checkpoints.append(backup) + raise OSError("focus failed after checkpoint corruption") + + monkeypatch.setattr(service._focus_service._config, "write_project_focus", fault) + with pytest.raises( + ImportExecutionError, match="focus rollback also failed.*checkpoint content changed" + ): + service.import_snapshot(dataclasses.replace(options, device=0)) + assert hashes(first.db_path) == old + assert checkpoints[0].read_bytes() == b"same inode, foreign checkpoint bytes" + assert (tmp_path / ".pt-snap/focus.json").read_bytes() != checkpoints[0].read_bytes() + + +@pytest.mark.parametrize("bad", ["empty", "corrupt", "absent-device", "alias-parent"]) +def test_dataset_input_and_output_prerequisites(tmp_path, bad): + options = source_options(tmp_path) + if bad == "empty": + from tests.snapshot.test_sharded_replay import write_source + + options = dataclasses.replace( + options, snapshot_file=write_source(tmp_path, {"segments": [], "device_traces": [[]]}) + ) + elif bad == "corrupt": + options.snapshot_file.write_bytes(b"not pickle") + elif bad == "absent-device": + options = dataclasses.replace(options, device=9) + elif bad == "alias-parent": + victim = tmp_path / "other-owner" + victim.mkdir() + alias = tmp_path / "alias" + alias.symlink_to(victim, target_is_directory=True) + options = dataclasses.replace(options, output_dir=alias / "nested") + with pytest.raises(ImportExecutionError): + ImportService().import_snapshot(options) + if bad == "alias-parent": + assert list(victim.iterdir()) == [] diff --git a/tests/test_dataset_import_cli.py b/tests/test_dataset_import_cli.py new file mode 100644 index 0000000..3b16fab --- /dev/null +++ b/tests/test_dataset_import_cli.py @@ -0,0 +1,81 @@ +import json +from pathlib import Path + +import pytest +from typer.testing import CliRunner + +from pt_snap_cli.api import SnapshotAnalyzer +from pt_snap_cli.cli import _safe_call, app +from tests.core.test_dataset_import import source_options + + +@pytest.fixture(autouse=True) +def isolate(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + monkeypatch.setenv("HOME", str(tmp_path)) + monkeypatch.setattr(Path, "home", lambda: tmp_path) + monkeypatch.delenv("PT_SNAP_DB_PATH", raising=False) + + +def test_dataset_cli_json_generation_reuse_force_focus_and_text(tmp_path): + options = source_options(tmp_path, multiple=True) + args = [ + "import", + str(options.snapshot_file), + "-o", + str(options.output_dir), + "--events-per-slice", + "2", + ] + runner = CliRunner() + result = runner.invoke(app, [*args, "--json"]) + assert result.exit_code == 0, result.output + data = json.loads(result.output) + assert data["ok"] is True + assert data["format"] == "pt-snap-native-v2" + assert data["devices"] == [0, 1] and data["slice_count"] == 10 + assert data["omitted_devices"] == [2] + assert data["dataset_path"] == data["db_path"] + assert data["rebuilt"] and not data["reused"] + assert data["focus_state"]["callstack_layout"] == "v2" + reused = runner.invoke(app, [*args, "--no-focus", "--json"]) + assert reused.exit_code == 0 + assert json.loads(reused.output)["reused"] is True + rebuilt = runner.invoke(app, [*args, "--force"]) + assert rebuilt.exit_code == 0 and "slices: 10" in rebuilt.output + focus = runner.invoke(app, ["focus", data["dataset_path"], "--device", "1", "--json"]) + assert focus.exit_code == 0 + focused = json.loads(focus.output) + with SnapshotAnalyzer(data["dataset_path"], device_id=1) as analyzer: + state = analyzer.get_focus() + assert state.db_path == focused["db_path"] + assert state.available_devices == focused["available_devices"] == [0, 1] + assert state.callstack_layout == focused["callstack_layout"] == "v2" + assert state.device_id == focused["device_id"] == 1 + + +@pytest.mark.parametrize( + "arguments", + [ + ["--events-per-slice", "0"], + ["--events-per-slice", "nope"], + ["--events-per-slice"], + ["--format", "unknown"], + ["--format", ""], + ["--format", "pt-snap-native-v2"], + ["--events-per-slice", "2", "--format", "single-db"], + ["--events-per-slice", "2", "--format", "compatibility-v1"], + ["--device", "-1"], + ], +) +def test_dataset_cli_argument_error_json(tmp_path, arguments, monkeypatch, capsys): + options = source_options(tmp_path) + monkeypatch.setattr( + "sys.argv", ["pt-snap", "import", str(options.snapshot_file), "--json", *arguments] + ) + assert _safe_call() != 0 + captured = capsys.readouterr() + assert captured.out == "" + data = json.loads(captured.err) + assert data["ok"] is False and data["error"]["code"] == "INVALID_PARAMETER" + assert not list(tmp_path.glob("*.pt-snap-native-v2")) diff --git a/tests/test_native_dataset.py b/tests/test_native_dataset.py new file mode 100644 index 0000000..bd341ef --- /dev/null +++ b/tests/test_native_dataset.py @@ -0,0 +1,212 @@ +import dataclasses +import json +import sqlite3 +from contextlib import closing +from pathlib import Path + +import pytest + +from pt_snap_cli.api import SnapshotAnalyzer +from pt_snap_cli.core.context_cache import ContextCache +from pt_snap_cli.core.dataset_contract import DatasetContractError, QueryScope, parse_manifest +from pt_snap_cli.core.dataset_resolver import DatasetResolver, _hash_file +from pt_snap_cli.core.errors import DatabaseSchemaError, InvalidParameterError, QueryExecutionError +from pt_snap_cli.core.import_service import ImportService +from pt_snap_cli.core.native_dataset_contract import parse_native_manifest +from tests.core.test_dataset_import import hashes, source_options + + +@pytest.fixture(autouse=True) +def isolate(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + monkeypatch.setenv("HOME", str(tmp_path)) + monkeypatch.setattr(Path, "home", lambda: tmp_path) + monkeypatch.delenv("PT_SNAP_DB_PATH", raising=False) + + +def dataset(tmp_path): + return ( + ImportService() + .import_snapshot( + dataclasses.replace(source_options(tmp_path, multiple=True), set_focus=False) + ) + .db_path + ) + + +def test_native_sparse_ids_positions_device_scope_borrowed_cache_and_readonly(tmp_path): + root = dataset(tmp_path) + before = hashes(root) + cache = ContextCache(maxsize=2) + try: + with SnapshotAnalyzer(root, context_cache=cache) as analyzer: + assert analyzer.get_database_metadata()["status"] == "available" + overview = analyzer.get_database_overview() + assert overview["devices"] == [ + {"device_id": d, "first_event_id": 7, "last_event_id": 87} for d in (0, 1) + ] + assert overview["dataset"]["real_event_count"] == 18 + assert overview["dataset"]["structured_frames"] is False + for device in (0, 1): + for position in range(9): + result = analyzer.execute_query( + "event", {"id": position * 10 + 7}, device_id=device + ) + assert result["rows"][0]["id"] == position * 10 + 7 + assert result["scope"]["slice_index"] == position // 2 + assert len(cache) <= 2 + assert len(analyzer.execute_query("event", {}, device_id=1, slice_index=0)["rows"]) == 2 + with pytest.raises(QueryExecutionError, match="cross-slice"): + analyzer.execute_query("memory_peak") + with pytest.raises(InvalidParameterError): + analyzer.execute_query("event", {"id": -1}) + held = cache.get(root / "device_1/slice_00000.db") + assert len(cache) > 0 + with held.connect() as conn: + assert conn.execute("SELECT 1").fetchone()[0] == 1 + resolved = DatasetResolver().inspect(root / "manifest.json") + assert len(resolved.paths(QueryScope("dataset"))) == 10 + assert ( + len(resolved.paths(QueryScope("event_range", 0, start_event_id=7, end_event_id=87))) + == 5 + ) + with pytest.raises(InvalidParameterError): + resolved.paths(QueryScope("event_range", 0, start_event_id=0, end_event_id=87)) + finally: + cache.close() + assert hashes(root) == before + assert not list(root.rglob("*-wal")) and not list(root.rglob("*-journal")) + raw = json.loads((root / "manifest.json").read_text()) + with pytest.raises(DatasetContractError): + parse_manifest(raw) # Never relabel native-v2 as compatibility-v1. + + +@pytest.mark.parametrize( + "part", + ["manifest", "device", "member", "wal", "journal", "shm", "dangling-wal", "persistent-wal"], +) +def test_native_alias_sidecar_and_persistent_wal_checks_precede_any_sqlite_open( + tmp_path, monkeypatch, part +): + root = dataset(tmp_path) + member = root / "device_1/slice_00004.db" + if part in ("manifest", "device", "member"): + path = {"manifest": root / "manifest.json", "device": root / "device_1", "member": member}[ + part + ] + original = tmp_path / ("original-" + part) + path.rename(original) + path.symlink_to(original, target_is_directory=part == "device") + elif part == "persistent-wal": + with closing(sqlite3.connect(member)) as conn: + assert conn.execute("PRAGMA journal_mode=WAL").fetchone() == ("wal",) + assert not Path(str(member) + "-wal").exists() + else: + sidecar = Path(str(member) + "-" + part.removeprefix("dangling-")) + if part.startswith("dangling"): + sidecar.symlink_to(tmp_path / "missing-sidecar") + else: + sidecar.write_bytes(b"live") + monkeypatch.setattr( + sqlite3, "connect", lambda *a, **k: pytest.fail("unsafe artifact opened SQLite") + ) + with pytest.raises(DatabaseSchemaError): + DatasetResolver().inspect(root) + + +@pytest.mark.parametrize( + "field,value", + [ + ("format", "compatibility-v1"), + ("schemaVersion", 99), + ("status", "building"), + ("omittedDevices", [0]), + ("omittedDevices", [2, 2]), + ], +) +def test_native_manifest_publication_and_version_validation(tmp_path, field, value): + root = dataset(tmp_path) + raw = json.loads((root / "manifest.json").read_text()) + raw[field] = value + with pytest.raises(DatasetContractError): + parse_native_manifest(raw) + + +@pytest.mark.parametrize( + "field,value", + [("eventCount", 89), ("sliceCount", True), ("readySlices", [0]), ("sliceCount", 3)], +) +def test_native_manifest_count_and_ready_validation(tmp_path, field, value): + root = dataset(tmp_path) + raw = json.loads((root / "manifest.json").read_text()) + raw["devices"]["0"][field] = value + with pytest.raises(DatasetContractError): + parse_native_manifest(raw) + + +@pytest.mark.parametrize( + "field,value", + [ + ("startPosition", 1), + ("endPosition", 999), + ("index", True), + ("startEventId", -1), + ("endEventId", 0), + ("file", "../escape.db"), + ("sha256", "bad"), + ("ready", False), + ], +) +def test_native_slice_validation(tmp_path, field, value): + root = dataset(tmp_path) + raw = json.loads((root / "manifest.json").read_text()) + raw["devices"]["0"]["slices"][0][field] = value + with pytest.raises(DatasetContractError): + parse_native_manifest(raw) + + +@pytest.mark.parametrize( + "sql", + [ + "UPDATE trace_entry_0 SET action=99 WHERE id=7", + "UPDATE block_0 SET id=id+1000 WHERE allocEventId>=0", + "DELETE FROM pt_snap_block_reference", + "UPDATE pt_snap_metadata SET source_name='wrong'", + "DELETE FROM dictionary", + "DROP TABLE callstack", + "UPDATE trace_entry_0 SET callstackId=9999 WHERE id=7", + ], +) +def test_full_native_validation_not_only_manifest_or_member_hash(tmp_path, sql): + root = dataset(tmp_path) + member = root / "device_0/slice_00000.db" + with closing(sqlite3.connect(member)) as conn, conn: + conn.execute(sql) + raw = json.loads((root / "manifest.json").read_text()) + raw["devices"]["0"]["slices"][0]["sha256"] = _hash_file(member) + (root / "manifest.json").write_text(json.dumps(raw)) + with pytest.raises(DatabaseSchemaError): + DatasetResolver().inspect(root) + + +def test_unknown_frame_extension_never_promotes_structured_capability_and_generation_changes( + tmp_path, +): + root = dataset(tmp_path) + cache = ContextCache() + try: + with SnapshotAnalyzer(root, context_cache=cache) as analyzer: + analyzer.execute_query("event", {"id": 7}) + old = next(iter(cache._entries.values()))[0] + old_connection = old._conn + raw = json.loads((root / "manifest.json").read_text()) + raw["extensions"] = {"structuredFrames": {"version": 999, "present": True}} + (root / "manifest.json").write_text(json.dumps(raw)) + analyzer.execute_query("event", {"id": 7}) + assert next(iter(cache._entries.values()))[0] is not old + assert analyzer.get_database_overview()["dataset"]["structured_frames"] is False + assert not hasattr(analyzer, "import_snapshot") + with pytest.raises(sqlite3.ProgrammingError): + old_connection.execute("SELECT 1") + finally: + cache.close()