From 67e26f4eb845541980a9493c2750060aaae1c0e0 Mon Sep 17 00:00:00 2001 From: codeasier Date: Mon, 5 Oct 2026 08:05:14 +0800 Subject: [PATCH 1/2] feat(dataset): define sharded SnapshotDB compatibility contract (#198) --- README.md | 1 + README_zh.md | 1 + docs/README.md | 2 + docs/en/sharded-snapshotdb.md | 198 +++++++++ docs/zh/sharded-snapshotdb.md | 162 ++++++++ src/pt_snap_cli/core/dataset_contract.py | 488 +++++++++++++++++++++++ tests/test_dataset_contract.py | 413 +++++++++++++++++++ 7 files changed, 1265 insertions(+) create mode 100644 docs/en/sharded-snapshotdb.md create mode 100644 docs/zh/sharded-snapshotdb.md create mode 100644 src/pt_snap_cli/core/dataset_contract.py create mode 100644 tests/test_dataset_contract.py diff --git a/README.md b/README.md index 2f28e83..5917953 100644 --- a/README.md +++ b/README.md @@ -98,6 +98,7 @@ See the [documentation index](docs/README.md) for all English and Chinese guides | Splitting snapshots | [Splitting Snapshots](docs/en/splitting.md) | | Agent skills | [Agent Skills](docs/en/skills.md) | | Database format | [SnapshotDB Schema](docs/en/database.md) | +| Sharded artifact contract (P0; not an exporter) | [Sharded SnapshotDB protocol](docs/en/sharded-snapshotdb.md) | | Python API | [SnapshotAnalyzer API](docs/en/snapshot-analyzer-api.md) | | Result mapping utility | [ResultMapper API](docs/en/result-mapper-api.md) | diff --git a/README_zh.md b/README_zh.md index 264544d..a37c593 100644 --- a/README_zh.md +++ b/README_zh.md @@ -91,6 +91,7 @@ helper 会先路由采集,无需先安装 CLI。分析仍需要 CLI 和 Snapsh | 拆分快照 | [拆分快照](docs/zh/splitting.md) | | Agent skill | [Agent Skills](docs/zh/skills.md) | | 数据库格式 | [SnapshotDB Schema](docs/zh/database.md) | +| 分片产物合同(P0,非导出器) | [分片 SnapshotDB 协议](docs/zh/sharded-snapshotdb.md) | | Python API | [SnapshotAnalyzer API](docs/zh/snapshot-analyzer-api.md) | | 结果映射工具 | [ResultMapper API](docs/zh/result-mapper-api.md) | diff --git a/docs/README.md b/docs/README.md index 619d4f1..f291838 100644 --- a/docs/README.md +++ b/docs/README.md @@ -12,6 +12,7 @@ | Splitting Snapshots | [en/splitting.md](en/splitting.md) | | Agent skills | [en/skills.md](en/skills.md) | | Database Schema | [en/database.md](en/database.md) | +| Sharded SnapshotDB protocol (P0) | [en/sharded-snapshotdb.md](en/sharded-snapshotdb.md) | | Import metadata | [en/database.md](en/database.md) | | SnapshotAnalyzer API | [en/snapshot-analyzer-api.md](en/snapshot-analyzer-api.md) | | ResultMapper API | [en/result-mapper-api.md](en/result-mapper-api.md) | @@ -28,6 +29,7 @@ | 拆分快照 | [zh/splitting.md](zh/splitting.md) | | Agent skill | [zh/skills.md](zh/skills.md) | | 数据库格式 | [zh/database.md](zh/database.md) | +| 分片 SnapshotDB 协议(P0) | [zh/sharded-snapshotdb.md](zh/sharded-snapshotdb.md) | | 导入 metadata | [zh/database.md](zh/database.md) | | SnapshotAnalyzer API | [zh/snapshot-analyzer-api.md](zh/snapshot-analyzer-api.md) | | ResultMapper API | [zh/result-mapper-api.md](zh/result-mapper-api.md) | diff --git a/docs/en/sharded-snapshotdb.md b/docs/en/sharded-snapshotdb.md new file mode 100644 index 0000000..5c45d1d --- /dev/null +++ b/docs/en/sharded-snapshotdb.md @@ -0,0 +1,198 @@ +# Sharded SnapshotDB protocol (P0, compatibility v1) + +[中文](../zh/sharded-snapshotdb.md) | English + +## Scope and versions + +This freezes an artifact contract and shared validation models, **not a new CLI +exporter or cross-slice query engine**. Existing `import` still defaults to one +native v2 DB; `split` still produces replayable pickle/JSON. Existing single-DB +v1/v2 analysis and focus are unchanged. The `complete` gate below applies only to +this new dataset contract, not to legacy single-DB analysis. + +`pt_snap_cli.core.dataset_contract` provides `parse_manifest`, +`validate_event_ids`, `validate_dataset`, immutable manifest/device/slice models, +`QueryScope`, and `DatasetContractError` (`location`, `code`). Producers can share +the same prepublication contract with readers. Validation never loads pickle, +creates missing databases, migrates SQLite, or writes focus. + +Separate version axes: + +- `schemaVersion=1`: manifest and compatibility base-table contract defined here. +- `pt_snap_metadata.import_format_version`: existing pt-snap importer format, + currently 2; **not** the manifest version and not required for external artifacts. +- Future structured-frame extensions have their own version. Unknown extensions + do not invalidate base reading or imply structured-frame support. + +## Layout and manifest + +The producer convention is `.msinsight/manifest.json` +with `device_/slice_.db` relative to that artifact directory. +An explicitly supplied relocated directory can be validated; `sourceFile` records +origin, not a path to open. Never load the source pickle during inspection. + +```json +{ + "schemaVersion": 1, + "status": "complete", + "sourceFile": "/capture/snapshot.pkl", + "cacheHash": "producer-specific-opaque-identity", + "eventsPerSlice": 2, + "devices": { + "0": { + "eventCount": 4, + "sliceCount": 2, + "readySlices": [0, 1], + "slices": [ + {"index": 0, "startEventId": 0, "endEventId": 1, "file": "device_0/slice_00000.db", "ready": true}, + {"index": 1, "startEventId": 2, "endEventId": 3, "file": "device_0/slice_00001.db", "ready": true} + ] + } + } +} +``` + +All fields shown are required. Integers are not booleans/floats; counts/ranges +fit signed 64-bit integers, version/index/sliceCount/device IDs fit nonnegative +signed 32-bit integers. Device keys are canonical decimal IDs (`0`, not `00`). +`sourceFile` is nonempty. `cacheHash` is an opaque producer token, possibly empty +for an external artifact: it is **not** assumed to be SHA-256 or proof of source +content. A future cache must distinguish source content and producer/format, +base/extension versions, selected devices, capacity, and finalized generation; +source path, size/mtime, `cacheHash` alone, or a per-slice index is insufficient. +This P0 module performs no cache reuse. + +`building` permits partial readiness for `parse_manifest(..., +require_complete=False)` inspection only. It still declares the complete planned +range layout. `complete` requires all slices ready **after identity/lifecycle +backfills finish**. Readiness indices must be unique, in range, and agree with each +slice's boolean. `validate_dataset` accepts only complete datasets with every +file present. Missing files are not created. Duplicate JSON members are rejected. + +Per device, slices are in index order starting at 0; inclusive ranges continuously +cover `0..eventCount-1` without overlap/gaps. Each range has at most +`eventsPerSlice` real events (short intermediate slices are legal). Count real +positions, not `max(id)+1`. Compatibility v1 **rejects nonzero-start, sparse, +reordered or duplicate original real IDs**; no silent renumbering. Empty devices +and static-only devices cannot be represented: reject a requested such device, +never invent event 0. An all-empty dataset is rejected. Other devices may be +explicitly selected, but omission must be disclosed by the producer. + +Paths must be exactly the declared device/index relative path: no absolute/drive/ +UNC paths, `..`, `.`, empty components, backslashes, NUL, device mismatch, aliases +or symlink members/parents. Validation uses read-only SQLite (`mode=ro`) and closes +all handles, including failures. These checks are not an OS sandbox or a +concurrent-publication lock. + +## Fixed base SQLite schema + +Each slice has **actual tables**, not compatibility views, for exactly its device: +`trace_entry_`, `block_`, and `dictionary`. Their column order is normative +because the upstream reader uses positional `SELECT *`. Native v2 `callstackId` +is not a compatible substitute for inline text. + +| Table | Columns, in order (SQLite declared type) | +| --- | --- | +| `trace_entry_` | `id INTEGER PRIMARY KEY`, `action INTEGER`, `address INTEGER`, `size INTEGER`, `stream INTEGER`, `allocated INTEGER`, `active INTEGER`, `reserved INTEGER`, `callstack TEXT` | +| `block_` | `id INTEGER PRIMARY KEY`, `address INTEGER`, `size INTEGER`, `requestedSize INTEGER`, `state INTEGER`, `allocEventId INTEGER`, `freeEventId INTEGER` | +| `dictionary` | `table TEXT`, `column TEXT`, `key TEXT`, `value TEXT` | + +The dictionary stores decimal integer keys as text, with the actual suffixed +table name and column name. Each slice includes all these exact mappings: + +| Column | Key → value | +| --- | --- | +| `trace_entry_.action` | `0 → segment_map`, `1 → segment_unmap`, `2 → segment_alloc`, `3 → segment_free`, `4 → alloc`, `5 → free_requested`, `6 → free_completed`, `7 → workspace_snapshot` | +| `block_.state` | `-1 → inactive`, `0 → active_pending_free`, `1 → active_allocated` | + +No unknown base action/state is accepted. In particular, native pt-snap's `oom=8` +is outside this pinned compatibility baseline; a future exporter must reject it +or define a separately negotiated extension, not relabel it. Metrics and sizes +are bytes; event IDs are chronological identities, **not timestamps**. + +## Boundary events and block lifetimes + +- Real events have nonnegative original IDs. `eventCount`, capacity, ranges and + real-event peaks count/filter only `id >= 0`. +- Negative trace IDs are separately identified synthetic left-boundary state rows + (`segment_map=0` or `segment_alloc=2`). They may have large memory totals but + must not compete for a real-event peak or consume capacity. IDs are slice-local; + `(device, slice, negative ID)` identifies a boundary row. Repeated boundary + segments do not represent new real allocations. Validation reports + `real_event_count` and `boundary_event_count` separately; it does not execute peaks. +- Resolved block identity is `(device ID, original allocation event ID)`; + `block.id == allocEventId >= 0` stays stable across every slice containing it. + Reusing an address after a free creates a **different** block. Never deduplicate + by address or by a slice-local row number. +- Preexisting/unresolved blocks have stable, dataset/device-local negative + `block.id` values and `allocEventId=-1`. Repeated appearances keep their ID; + distinct objects at the same address must not collide. Negative identity is + not a known allocation event; a future enhanced identity/coverage table may + distinguish preexisting from other unresolved cases. +- `freeEventId=-1` means no observed free (live **or unknown**, not proof of a leak). + Otherwise it is a real ID in the device trace, not earlier than a known alloc. + `allocEventId=-1` means unknown/preexisting, not event 0. Other negative + lifecycle sentinels are invalid. Full finalized IDs/lifetimes may point outside + a slice's range. Slice observations may differ in state, but repeated IDs must + agree on address, size, requested size, alloc/free IDs. Finalization cannot + leave temporary placeholder IDs for a resolved allocation. + +The base format cannot prove that two colliding unresolved records are distinct +if a producer has discarded their identity; correct producer tracking is required. + +## Extensions, scopes and capabilities + +Keep optional metadata, deduplicated stacks and ordered frames in isolated +`pt_snap_*` extension tables or sidecars, never append columns to positional base +tables. In particular, **do not attach an unprefixed native `callstack` table to +inline v1**: the existing pt-snap layout detector correctly considers it a v1/v2 +conflict. Native v2 remains a separate single-DB layout, not this compatibility +base. `pt_snap_metadata` and auxiliary reference tables are optional here. + +Unknown `extensions` declarations/tables are ignored by base validation. +`DatasetValidation.text_callstacks=True` means an inline text column is available, +not that every event has a stack. `structured_frames=False` remains false even +when a manifest claims frames: complete structured-frame capability requires a +recognized extension version, ordered frame rows, valid references and coverage, +which P0 does not implement. Text stacks cannot substitute for these guarantees. + +`QueryScope.validate(manifest)` defines selectors, not execution support: + +| Scope | Required selectors and extent | +| --- | --- | +| `dataset` | No selectors; all declared devices and their full real traces | +| `device` | `device_id`; one declared device's full real trace | +| `slice` | `device_id`, `slice_index`; exactly that slice's real interval, boundary separate | +| `event_range` | `device_id`, inclusive `start_event_id/end_event_id` within the real trace, possibly across slices | + +There is no global event order across devices. Boundary IDs cannot be range +endpoints. `query_execution=False` explicitly declares that cross-slice execution +is deferred; passing scope validation does not make existing CLI/API accept a +manifest or directory. + +```python +from pt_snap_cli.core.dataset_contract import QueryScope, validate_dataset + +inspection = validate_dataset("/capture/snapshot.pkl.msinsight") +QueryScope("event_range", device_id=0, start_event_id=1, end_event_id=3).validate( + inspection.manifest +) +assert not inspection.structured_frames +``` + +## Evidence and limits + +Interoperability facts were read from fixed upstream +[manifest validation](https://github.com/Ascend/msinsight/blob/101f65b877a267ffd5f66ea3834706057ba243e5/server/src/modules/memsnapshot/service/MemSnapshotSliceService.cpp), +[positional SQLite reading](https://github.com/Ascend/msinsight/blob/101f65b877a267ffd5f66ea3834706057ba243e5/server/src/modules/memsnapshot/database/MemSnapshotDatabase.cpp), +[base schemas](https://github.com/Ascend/msinsight/blob/101f65b877a267ffd5f66ea3834706057ba243e5/scripts/MemSnapDump/tools/adaptors/database/snapshot_db.py), +and [publication/backfills](https://github.com/Ascend/msinsight/blob/101f65b877a267ffd5f66ea3834706057ba243e5/scripts/MemSnapDump/tools/adaptors/snapshot2db.py). +Those source files carry Huawei's Mulan PSL v2 notice. This contract, Python +validator and synthetic tests are original local implementation of interface +facts; no upstream implementation is copied into the package. The upstream docs +license is separately CC BY 4.0, not a replacement for those source notices. + +Our complete-readiness and canonical-path rules are intentionally stricter than +the upstream parser (which also supports building). Synthetic contract tests and +source-function comparisons are not GUI acceptance, native exporter acceptance, +performance measurements, or proof of interoperability with a different revision. diff --git a/docs/zh/sharded-snapshotdb.md b/docs/zh/sharded-snapshotdb.md new file mode 100644 index 0000000..36652da --- /dev/null +++ b/docs/zh/sharded-snapshotdb.md @@ -0,0 +1,162 @@ +# 分片 SnapshotDB 协议(P0,兼容 v1) + +中文 | [English](../en/sharded-snapshotdb.md) + +## 范围与版本 + +本阶段冻结产物协议和共享校验模型,**不新增 CLI 导出器或跨片查询引擎**。现有 +`import` 默认仍生成一个原生 v2 DB;`split` 仍输出可回放 pickle/JSON。现有单库 +v1/v2 分析与 focus 不变。下述 `complete` 门槛只约束新数据集合同,不约束旧单库分析。 + +`pt_snap_cli.core.dataset_contract` 提供 `parse_manifest`、`validate_event_ids`、 +`validate_dataset`、不可变 manifest/device/slice 模型、`QueryScope` 和带 +`location/code` 的 `DatasetContractError`。生产端可在发布前与读取端共用同一合同。 +校验不加载 pickle、不创建缺失数据库、不迁移 SQLite、不写 focus。 + +版本轴分别管理: + +- `schemaVersion=1`:本文的 manifest 和兼容基础表协议。 +- `pt_snap_metadata.import_format_version`:已有 pt-snap 导入格式,当前为 2; + **不是** manifest 版本,外部产物不要求该表。 +- 后续结构化 frame 扩展独立版本化。未知扩展不影响基础读取,也不意味着支持结构化 frames。 + +## 目录与 manifest + +生产端布局为 `<原始 snapshot 路径>.msinsight/manifest.json`,分片相对于该目录放在 +`device_/slice_.db`。允许显式指定已搬迁的产物目录进行校验; +`sourceFile` 记录来源,不是要打开的路径。检查时不得加载源 pickle。 + +```json +{ + "schemaVersion": 1, + "status": "complete", + "sourceFile": "/capture/snapshot.pkl", + "cacheHash": "producer-specific-opaque-identity", + "eventsPerSlice": 2, + "devices": { + "0": { + "eventCount": 4, + "sliceCount": 2, + "readySlices": [0, 1], + "slices": [ + {"index": 0, "startEventId": 0, "endEventId": 1, "file": "device_0/slice_00000.db", "ready": true}, + {"index": 1, "startEventId": 2, "endEventId": 3, "file": "device_0/slice_00001.db", "ready": true} + ] + } + } +} +``` + +示例中所有字段必需。整数不能是布尔或浮点数;计数/区间使用有符号 64 位整数, +版本/index/sliceCount/device ID 使用非负有符号 32 位整数。设备 key 为规范十进制 +ID(`0` 而非 `00`)。`sourceFile` 非空。`cacheHash` 是生产端不透明 token, +外部产物可为空:**不能**假定它是 SHA-256 或源内容证据。未来缓存需要区分源内容、 +生产端/格式、基础/扩展版本、所选设备、容量和最终化 generation;单独的源路径、 +size/mtime、`cacheHash` 或单片 index 都不足以复用。本 P0 模块不执行缓存复用。 + +`building` 只允许用 `parse_manifest(..., require_complete=False)` 检查部分就绪状态, +仍须声明完整计划区间。`complete` 必须在**身份/生命周期回填结束后**使所有片就绪。 +就绪 index 不重复、不越界,并与各片布尔值一致。`validate_dataset` 只接受所有文件 +齐全的 complete 数据集;不会创建缺片。重复 JSON 成员被拒绝。 + +各设备分片从 index 0 顺序排列;闭区间无重叠/缺口地连续覆盖 `0..eventCount-1`。 +每片最多 `eventsPerSlice` 个真实事件(允许中间片较短)。按真实事件位置计数,不能 +用 `max(id)+1` 推算。兼容 v1 **拒绝非零起点、稀疏、乱序或重复的原始真实 ID**, +不得静默改号。空设备和仅有静态 block 的设备无法表示:请求此类设备时明确拒绝, +不得虚构 event 0;全部为空的数据集被拒绝。可明确选择其他设备,但生产端必须披露省略。 + +路径须严格对应声明的 device/index 相对路径;拒绝绝对/盘符/UNC 路径、`..`、`.`、 +空分量、反斜线、NUL、设备错配、别名及成员/父目录符号链接。校验以 `mode=ro` +打开 SQLite,包括失败路径都关闭资源。这不是 OS 沙箱,也不锁住并发发布。 + +## 固定基础 SQLite schema + +每片须有本设备的**实际表**,不能用兼容 VIEW 替代:`trace_entry_`、`block_`、 +`dictionary`。上游部分 `SELECT *` 按位置读取,因此列序属于合同。原生 v2 的 +`callstackId` 不能替代内联文本。 + +| 表 | 按顺序排列的列(SQLite 声明类型) | +| --- | --- | +| `trace_entry_` | `id INTEGER PRIMARY KEY`, `action INTEGER`, `address INTEGER`, `size INTEGER`, `stream INTEGER`, `allocated INTEGER`, `active INTEGER`, `reserved INTEGER`, `callstack TEXT` | +| `block_` | `id INTEGER PRIMARY KEY`, `address INTEGER`, `size INTEGER`, `requestedSize INTEGER`, `state INTEGER`, `allocEventId INTEGER`, `freeEventId INTEGER` | +| `dictionary` | `table TEXT`, `column TEXT`, `key TEXT`, `value TEXT` | + +dictionary 的十进制整数 key 以文本存储,使用真实带设备后缀的表名与列名。每片包含 +以下完整且精确的映射: + +| 列 | key → value | +| --- | --- | +| `trace_entry_.action` | `0 → segment_map`, `1 → segment_unmap`, `2 → segment_alloc`, `3 → segment_free`, `4 → alloc`, `5 → free_requested`, `6 → free_completed`, `7 → workspace_snapshot` | +| `block_.state` | `-1 → inactive`, `0 → active_pending_free`, `1 → active_allocated` | + +不接受未知基础 action/state。特别是原生 pt-snap 的 `oom=8` 不在固定版本兼容基线内; +未来导出器须拒绝或协商独立扩展,不得重标记。指标/大小单位为字节;event ID 是按时间 +排序的身份,**不是时间戳**。 + +## 边界事件与 block 生命周期 + +- 真实事件保留非负原始 ID。`eventCount`、容量、范围及真实事件峰值只计数/筛选 `id >= 0`。 +- 负 trace ID 独立标识合成的左边界状态行(`segment_map=0` 或 `segment_alloc=2`)。 + 即使内存总量很大,也不能竞争真实峰值或占容量。ID 在片内局部,使用 + `(device, slice, 负 ID)` 标识边界行;重复边界 segment 不等于新真实分配。 + 校验分别报告 `real_event_count/boundary_event_count`,不执行峰值查询。 +- 已解析 block 的身份是 `(device ID, 原始 allocation event ID)`;所有出现片中 + `block.id == allocEventId >= 0`。释放后同地址复用产生**另一个** block。 + 禁止按地址或片内行号去重。 +- 预存/未解析 block 使用数据集/设备内稳定的负 `block.id`,`allocEventId=-1`。 + 重复出现保留 ID,同地址不同对象不得碰撞。负身份不是已知分配事件;未来增强身份/覆盖表 + 可区分预存和其他未解析情况。 +- `freeEventId=-1` 表示没有观察到释放(存活**或未知**,不证明泄漏);否则必须是设备内 + 真实 ID,不能早于已知 alloc。`allocEventId=-1` 表示未知/预存,不是 event 0; + 其他负生命周期哨兵非法。最终化的完整 ID/生命周期可指向当前片范围外。各片状态观察可 + 不同,但重复身份的地址、大小、requested size、alloc/free ID 必须一致。已解析分配 + 最终化后不得遗留临时占位 ID。 + +若生产端已丢失身份,基础格式无法证明两个碰撞的未解析记录是不同对象;生产端必须正确跟踪。 + +## 扩展、scope 与能力 + +可选 metadata、去重 stack、有序 frame 使用隔离的 `pt_snap_*` 扩展表或 sidecar, +不能向按位置读取的基础表追加列。尤其**不能给内联 v1 附加未命名隔离的原生 +`callstack` 表**:现有布局检测会正确报告 v1/v2 冲突。原生 v2 仍是独立单库布局, +不是本兼容基础格式。这里不要求 `pt_snap_metadata` 或辅助引用表。 + +基础校验忽略未知 `extensions` 声明/表。`DatasetValidation.text_callstacks=True` +表示有内联文本列,不保证每个事件有栈。即使 manifest 声称 frames, +`structured_frames=False` 仍为假:完整结构化 frame 能力要求已识别的独立扩展版本、 +有序 frame 行、合法引用及覆盖率,本 P0 不实现。文本栈不能代替这些保证。 + +`QueryScope.validate(manifest)` 只定义 selector,不宣称支持执行: + +| scope | 必需 selector 与范围 | +| --- | --- | +| `dataset` | 无 selector;所有声明设备的完整真实 trace | +| `device` | `device_id`;一个声明设备的完整真实 trace | +| `slice` | `device_id`, `slice_index`;该片真实区间,边界独立 | +| `event_range` | `device_id`、设备内闭区间 `start_event_id/end_event_id`;可跨片 | + +设备间没有全局事件顺序。边界 ID 不能作为范围端点。`query_execution=False` +明确表示跨片执行留后续;scope 校验成功不使现有 CLI/API 自动接受 manifest 或目录。 + +```python +from pt_snap_cli.core.dataset_contract import QueryScope, validate_dataset + +inspection = validate_dataset("/capture/snapshot.pkl.msinsight") +QueryScope("event_range", device_id=0, start_event_id=1, end_event_id=3).validate( + inspection.manifest +) +assert not inspection.structured_frames +``` + +## 来源证据与限制 + +接口事实已从固定版本真实源码读取: +[manifest 校验](https://github.com/Ascend/msinsight/blob/101f65b877a267ffd5f66ea3834706057ba243e5/server/src/modules/memsnapshot/service/MemSnapshotSliceService.cpp)、 +[SQLite 按位置读取](https://github.com/Ascend/msinsight/blob/101f65b877a267ffd5f66ea3834706057ba243e5/server/src/modules/memsnapshot/database/MemSnapshotDatabase.cpp)、 +[基础 schema](https://github.com/Ascend/msinsight/blob/101f65b877a267ffd5f66ea3834706057ba243e5/scripts/MemSnapDump/tools/adaptors/database/snapshot_db.py)、 +[发布/回填](https://github.com/Ascend/msinsight/blob/101f65b877a267ffd5f66ea3834706057ba243e5/scripts/MemSnapDump/tools/adaptors/snapshot2db.py)。 +这些源码包含 Huawei Mulan PSL v2 声明。本文合同、Python 校验器和合成测试是本地原创的 +接口事实实现,没有向包内复制上游实现;上游 docs 单独使用 CC BY 4.0,不能替代源码声明。 + +complete 就绪一致性和规范路径规则刻意严于上游 parser(上游也支持 building)。 +合成合同测试/真实源函数对照不是 GUI 验收、原生导出器验收、性能测量,也不证明其他版本互通。 diff --git a/src/pt_snap_cli/core/dataset_contract.py b/src/pt_snap_cli/core/dataset_contract.py new file mode 100644 index 0000000..d863c1e --- /dev/null +++ b/src/pt_snap_cli/core/dataset_contract.py @@ -0,0 +1,488 @@ +"""P0 sharded SnapshotDB contract, independent of import/query execution. + +Original pt-snap implementation of the documented interoperability facts at +Ascend/msinsight revision 101f65b877a267ffd5f66ea3834706057ba243e5. +No upstream implementation is copied. See docs/en/sharded-snapshotdb.md. +""" + +from __future__ import annotations + +import json +import re +import sqlite3 +from collections.abc import Iterable +from contextlib import closing +from dataclasses import dataclass +from pathlib import Path +from typing import Literal, cast + +MANIFEST_SCHEMA_VERSION = 1 +MSINSIGHT_REVISION = "101f65b877a267ffd5f66ea3834706057ba243e5" +ACTION_NAMES = ( + "segment_map", + "segment_unmap", + "segment_alloc", + "segment_free", + "alloc", + "free_requested", + "free_completed", + "workspace_snapshot", +) +BLOCK_STATES = ((-1, "inactive"), (0, "active_pending_free"), (1, "active_allocated")) +TRACE_COLUMNS = tuple( + (name, "INTEGER") + for name in ("id", "action", "address", "size", "stream", "allocated", "active", "reserved") +) + (("callstack", "TEXT"),) +BLOCK_COLUMNS = tuple( + (name, "INTEGER") + for name in ("id", "address", "size", "requestedSize", "state", "allocEventId", "freeEventId") +) +DICTIONARY_COLUMNS = tuple((name, "TEXT") for name in ("table", "column", "key", "value")) + + +class DatasetContractError(ValueError): + """A validation failure with a stable code and manifest/file location.""" + + def __init__(self, location: str, code: str, detail: str): + self.location = location + self.code = code + super().__init__(f"{location}: {code}: {detail}") + + +def _need(condition: bool, location: str, code: str, detail: str) -> None: + if not condition: + raise DatasetContractError(location, code, detail) + + +def _object(value: object, location: str) -> dict[str, object]: + _need(isinstance(value, dict), location, "type", "expected object") + return cast(dict[str, object], value) + + +def _array(value: object, location: str) -> list[object]: + _need(isinstance(value, list), location, "type", "expected array") + return cast(list[object], value) + + +def _integer(value: object, location: str, minimum: int = 0, maximum: int = 2**63 - 1) -> int: + _need(type(value) is int, location, "type", "expected integer (not boolean)") + number = cast(int, value) + _need(minimum <= number <= maximum, location, "range", "integer out of range") + return number + + +def _text(value: object, location: str, *, empty: bool = False) -> str: + _need(isinstance(value, str), location, "type", "expected string") + text = cast(str, value) + _need(empty or bool(text), location, "range", "string must not be empty") + return text + + +@dataclass(frozen=True) +class SliceRecord: + index: int + start_event_id: int + end_event_id: int + file: str + ready: bool + + +@dataclass(frozen=True) +class DeviceRecord: + device_id: int + event_count: int + ready_slices: tuple[int, ...] + slices: tuple[SliceRecord, ...] + + +@dataclass(frozen=True) +class DatasetManifest: + schema_version: int + status: str + source_file: str + cache_hash: str + events_per_slice: int + devices: tuple[DeviceRecord, ...] + + def require_complete(self) -> None: + _need(self.status == "complete", "status", "building", "complete dataset required") + for device in self.devices: + _need( + all(item.ready for item in device.slices), + f"devices.{device.device_id}.readySlices", + "not_ready", + "complete requires every slice ready after finalization", + ) + + +def parse_manifest(value: object, *, require_complete: bool = True) -> DatasetManifest: + """Validate JSON-shaped data; inspection may explicitly accept building. + + Unknown extension declarations are ignored, never promoted to capabilities. + This does not check files; use validate_dataset for finalized artifacts. + """ + data = _object(value, "manifest") + version = _integer(data.get("schemaVersion"), "schemaVersion", maximum=2**31 - 1) + _need( + version == MANIFEST_SCHEMA_VERSION, "schemaVersion", "version", "unsupported base version" + ) + status = _text(data.get("status"), "status") + _need(status in ("building", "complete"), "status", "status", "unknown publication state") + source = _text(data.get("sourceFile"), "sourceFile") + cache_hash = _text(data.get("cacheHash"), "cacheHash", empty=True) + capacity = _integer(data.get("eventsPerSlice"), "eventsPerSlice", minimum=1) + devices_data = _object(data.get("devices"), "devices") + _need(bool(devices_data), "devices", "empty", "at least one event-bearing device required") + devices: list[DeviceRecord] = [] + for key, value in devices_data.items(): + location = f"devices.{key}" + _need( + isinstance(key, str) and re.fullmatch(r"0|[1-9][0-9]*", key) is not None, + location, + "device", + "expected canonical nonnegative device ID", + ) + device_id = _integer(int(key), location, maximum=2**31 - 1) + device = _object(value, location) + count = _integer(device.get("eventCount"), location + ".eventCount", minimum=1) + size = _integer( + device.get("sliceCount"), location + ".sliceCount", minimum=1, maximum=2**31 - 1 + ) + ready = tuple( + _integer(item, location + ".readySlices", maximum=size - 1) + for item in _array(device.get("readySlices"), location + ".readySlices") + ) + _need( + len(set(ready)) == len(ready), + location + ".readySlices", + "duplicate", + "duplicate ready index", + ) + records = _array(device.get("slices"), location + ".slices") + _need( + len(records) == size, + location + ".sliceCount", + "count", + "sliceCount differs from slices", + ) + slices: list[SliceRecord] = [] + expected_start = 0 + for index, raw in enumerate(records): + loc = f"{location}.slices[{index}]" + item = _object(raw, loc) + number = _integer(item.get("index"), loc + ".index", maximum=2**31 - 1) + _need(number == index, loc + ".index", "index", "indices must match array order") + start = _integer(item.get("startEventId"), loc + ".startEventId") + end = _integer(item.get("endEventId"), loc + ".endEventId", minimum=start) + _need( + start == expected_start, loc, "coverage", "ranges must cover continuously from zero" + ) + _need(end - start + 1 <= capacity, loc, "capacity", "slice exceeds eventsPerSlice") + file = _text(item.get("file"), loc + ".file") + _need( + not any(part in ("", ".", "..") for part in file.split("/")) + and not any(char in file for char in ("\\", ":", "\0")), + loc + ".file", + "path", + "unsafe relative path", + ) + _need( + file == f"device_{device_id}/slice_{index:05d}.db", + loc + ".file", + "path", + "file must match device and slice index", + ) + is_ready = item.get("ready") + _need(type(is_ready) is bool, loc + ".ready", "type", "expected boolean") + _need( + is_ready == (index in ready), + loc + ".ready", + "readiness", + "ready disagrees with readySlices", + ) + slices.append(SliceRecord(index, start, end, file, cast(bool, is_ready))) + expected_start = end + 1 + _need( + expected_start == count, + location + ".eventCount", + "coverage", + "ranges do not cover eventCount", + ) + devices.append(DeviceRecord(device_id, count, ready, tuple(slices))) + manifest = DatasetManifest(version, status, source, cache_hash, capacity, tuple(devices)) + # Even inspection must reject a contradictory complete publication. + if require_complete or status == "complete": + manifest.require_complete() + return manifest + + +def validate_event_ids(event_ids: Iterable[int]) -> int: + """Producer precondition: actual real-event positions, not max(id)+1. + + Boundary rows are a separate input category. No renumbering is performed. + """ + count = 0 + for position, value in enumerate(event_ids): + number = _integer(value, f"events[{position}]") + _need( + number == position, + f"events[{position}]", + "event_identity", + "compatibility requires original real IDs 0..N-1; do not silently renumber", + ) + count += 1 + _need(count > 0, "events", "empty", "empty/static-only devices cannot use compatibility v1") + return count + + +@dataclass(frozen=True) +class QueryScope: + """Selector contract only; not a cross-slice query executor.""" + + kind: Literal["dataset", "device", "slice", "event_range"] + device_id: int | None = None + slice_index: int | None = None + start_event_id: int | None = None + end_event_id: int | None = None + + def validate(self, manifest: DatasetManifest) -> None: + manifest.require_complete() + loc = "scope" + _need( + self.kind in ("dataset", "device", "slice", "event_range"), loc, "kind", "unknown scope" + ) + if self.kind == "dataset": + _need( + all( + value is None + for value in ( + self.device_id, + self.slice_index, + self.start_event_id, + self.end_event_id, + ) + ), + loc, + "selector", + "dataset has no selectors", + ) + return + device_id = _integer(self.device_id, loc + ".device_id", maximum=2**31 - 1) + device = next((item for item in manifest.devices if item.device_id == device_id), None) + _need(device is not None, loc, "device", "unknown device") + assert device is not None + if self.kind == "slice": + _integer(self.slice_index, loc + ".slice_index", maximum=len(device.slices) - 1) + else: + _need( + self.slice_index is None, loc, "selector", "slice index only valid for slice scope" + ) + if self.kind == "event_range": + start = _integer( + self.start_event_id, loc + ".start_event_id", maximum=device.event_count - 1 + ) + _integer( + self.end_event_id, + loc + ".end_event_id", + minimum=start, + maximum=device.event_count - 1, + ) + else: + _need( + self.start_event_id is None and self.end_event_id is None, + loc, + "selector", + "unexpected range", + ) + + +@dataclass(frozen=True) +class DatasetValidation: + manifest: DatasetManifest + real_event_count: int + boundary_event_count: int + text_callstacks: bool = True + structured_frames: bool = False + query_execution: bool = False + + +def _unique_object(pairs: list[tuple[str, object]]) -> dict[str, object]: + result: dict[str, object] = {} + for key, value in pairs: + _need(key not in result, "manifest." + key, "duplicate", "duplicate JSON member") + result[key] = value + return result + + +def _safe_file(root: Path, relative: str) -> Path: + path = root / relative + for part in ( + root, + *[ + root.joinpath(*Path(relative).parts[:i]) + for i in range(1, len(Path(relative).parts) + 1) + ], + ): + _need(not part.is_symlink(), relative, "path", "symlink is not an artifact member") + _need(path.is_file(), relative, "missing", "artifact file missing") + return path + + +def _schema( + conn: sqlite3.Connection, table: str, columns: tuple[tuple[str, str], ...], loc: str +) -> None: + kind = conn.execute("SELECT type FROM sqlite_master WHERE name=?", (table,)).fetchone() + _need(kind == ("table",), loc, "table", f"{table} must be an actual table, not a view") + rows = conn.execute(f'PRAGMA table_info("{table}")').fetchall() + _need( + tuple((row[1], row[2].upper()) for row in rows) == columns, + loc, + "columns", + f"{table} column order/types differ from base contract", + ) + if table != "dictionary": + _need( + tuple(row[5] for row in rows) == (1,) + (0,) * (len(rows) - 1), + loc, + "primary_key", + f"{table}.id must be the sole primary key", + ) + + +def _validate_slice( + conn: sqlite3.Connection, + device: DeviceRecord, + item: SliceRecord, + identities: dict[tuple[int, int], tuple[object, ...]], +) -> int: + loc = item.file + trace, block = f"trace_entry_{device.device_id}", f"block_{device.device_id}" + tables = { + row[0] + for row in conn.execute("SELECT name FROM sqlite_master WHERE type IN ('table','view')") + } + relevant = {name for name in tables if name.lower().startswith(("trace_entry_", "block_"))} + _need(relevant == {trace, block}, loc, "device_tables", "device/table set mismatch") + _need( + "callstack" not in {name.lower() for name in tables}, + loc, + "layout", + "v1 must not contain native v2 callstack table", + ) + for table, columns in ( + (trace, TRACE_COLUMNS), + (block, BLOCK_COLUMNS), + ("dictionary", DICTIONARY_COLUMNS), + ): + _schema(conn, table, columns, loc) + for table, column, values in ( + (trace, "action", tuple(enumerate(ACTION_NAMES))), + (block, "state", BLOCK_STATES), + ): + rows = conn.execute( + 'SELECT "key", "value" FROM dictionary WHERE "table"=? AND "column"=?', (table, column) + ).fetchall() + expected = {(str(key), name) for key, name in values} + _need( + len(rows) == len(expected) and set(rows) == expected, + loc, + "dictionary", + f"invalid {table}.{column} mapping", + ) + count, low, high = conn.execute( + f'SELECT COUNT(*), MIN(id), MAX(id) FROM "{trace}" WHERE id>=0' + ).fetchone() + _need( + (count, low, high) + == (item.end_event_id - item.start_event_id + 1, item.start_event_id, item.end_event_id), + loc, + "event_identity", + "real rows differ from declared contiguous interval", + ) + bad = conn.execute( + f'SELECT id FROM "{trace}" WHERE action NOT BETWEEN 0 AND 7 OR action IS NULL ' + "OR typeof(action) != 'integer' OR (id<0 AND action NOT IN (0,2)) LIMIT 1" + ).fetchone() + _need(bad is None, loc, "event_action", "unknown action or invalid boundary event") + invalid_types = " OR ".join(f"typeof({name}) != 'integer'" for name, _ in TRACE_COLUMNS[:-1]) + bad = conn.execute( + f'SELECT id FROM "{trace}" WHERE {invalid_types} ' + "OR size<0 OR allocated<0 OR active<0 OR reserved<0 " + "OR typeof(callstack) NOT IN ('text','null') LIMIT 1" + ).fetchone() + _need(bad is None, loc, "event_values", "invalid scalar type or negative byte metric") + boundary_count = conn.execute(f'SELECT COUNT(*) FROM "{trace}" WHERE id<0').fetchone()[0] + for row in conn.execute(f'SELECT * FROM "{block}"'): + block_id, address, size, requested, state, alloc, free = row + for name, value in zip( + ("id", "address", "size", "requestedSize", "state", "allocEventId", "freeEventId"), + row, + strict=True, + ): + _integer(value, f"{loc}.{block}.{block_id}.{name}", minimum=-(2**63)) + _need( + state in (-1, 0, 1) and size >= 0 and 0 <= requested <= size, + loc, + "block", + "invalid state/size", + ) + _need( + (alloc == -1 and block_id < 0) + or (0 <= alloc < device.event_count and block_id == alloc), + loc, + "block_identity", + "resolved ID must equal allocation; unresolved ID must be negative", + ) + _need( + free == -1 or (0 <= free < device.event_count and (alloc == -1 or free >= alloc)), + loc, + "lifecycle", + "invalid free sentinel or lifecycle order", + ) + identity = (device.device_id, block_id) + # State describes the slice observation and may differ; identity/lifetime do not. + signature = (address, size, requested, alloc, free) + _need( + identity not in identities or identities[identity] == signature, + loc, + "block_identity", + "same cross-slice ID has conflicting identity/lifetime", + ) + identities[identity] = signature + return cast(int, boundary_count) + + +def validate_dataset(directory: Path | str) -> DatasetValidation: + """Read a complete compatibility-v1 dataset without migration or pickle. + + Checks are validation, not a filesystem sandbox or concurrent publication lock. + Missing pt_snap_metadata/extension tables are valid external base artifacts. + """ + root = Path(directory).absolute() + _need( + root.resolve() == root and not root.is_symlink(), + "dataset", + "path", + "canonical non-symlink directory required", + ) + try: + path = _safe_file(root, "manifest.json") + with path.open(encoding="utf-8") as source: + data: object = json.load(source, object_pairs_hook=_unique_object) + manifest = parse_manifest(data) + identities: dict[tuple[int, int], tuple[object, ...]] = {} + boundaries = 0 + for device in manifest.devices: + for item in device.slices: + db_path = _safe_file(root, item.file) + try: + with closing(sqlite3.connect(db_path.as_uri() + "?mode=ro", uri=True)) as conn: + conn.execute("PRAGMA query_only=ON") + boundaries += _validate_slice(conn, device, item, identities) + except sqlite3.DatabaseError as exc: + raise DatasetContractError(item.file, "sqlite", str(exc)) from exc + return DatasetValidation( + manifest, sum(item.event_count for item in manifest.devices), boundaries + ) + except (OSError, UnicodeError, json.JSONDecodeError) as exc: + raise DatasetContractError("manifest.json", "read", str(exc)) from exc diff --git a/tests/test_dataset_contract.py b/tests/test_dataset_contract.py new file mode 100644 index 0000000..2ceae85 --- /dev/null +++ b/tests/test_dataset_contract.py @@ -0,0 +1,413 @@ +"""Synthetic P0 contract tests; no executable snapshot fixtures or upstream code.""" + +import json +import sqlite3 +from contextlib import closing +from copy import deepcopy +from pathlib import Path + +import pytest + +from pt_snap_cli.context import Context +from pt_snap_cli.core.dataset_contract import ( + ACTION_NAMES, + BLOCK_STATES, + DatasetContractError, + QueryScope, + parse_manifest, + validate_dataset, + validate_event_ids, +) + + +def manifest_data(): + return { + "schemaVersion": 1, + "status": "complete", + "sourceFile": "/capture/snapshot.pkl", + "cacheHash": "producer-specific-opaque-identity", + "eventsPerSlice": 2, + "devices": { + "0": { + "eventCount": 4, + "sliceCount": 2, + "readySlices": [0, 1], + "slices": [ + { + "index": 0, + "startEventId": 0, + "endEventId": 1, + "file": "device_0/slice_00000.db", + "ready": True, + }, + { + "index": 1, + "startEventId": 2, + "endEventId": 3, + "file": "device_0/slice_00001.db", + "ready": True, + }, + ], + } + }, + } + + +def make_dataset(root): + root.mkdir() + data = manifest_data() + (root / "manifest.json").write_text(json.dumps(data), encoding="utf-8") + (root / "device_0").mkdir() + for index in range(2): + path = root / f"device_0/slice_{index:05d}.db" + with closing(sqlite3.connect(path)) as conn, conn: + # Independent fixed DDL catches changes to the validator's constants. + conn.executescript( + 'CREATE TABLE dictionary ("table" TEXT, "column" TEXT, "key" TEXT, "value" TEXT);' + "CREATE TABLE trace_entry_0 (id INTEGER PRIMARY KEY, action INTEGER, " + "address INTEGER, size INTEGER, stream INTEGER, allocated INTEGER, " + "active INTEGER, reserved INTEGER, callstack TEXT);" + "CREATE TABLE block_0 (id INTEGER PRIMARY KEY, address INTEGER, size INTEGER, " + "requestedSize INTEGER, state INTEGER, allocEventId INTEGER, freeEventId INTEGER);" + ) + conn.executemany( + "INSERT INTO dictionary VALUES (?, ?, ?, ?)", + [ + ("trace_entry_0", "action", str(key), value) + for key, value in enumerate(ACTION_NAMES) + ] + + [("block_0", "state", str(key), value) for key, value in BLOCK_STATES], + ) + for event_id in range(index * 2, index * 2 + 2): + conn.execute( + "INSERT INTO trace_entry_0 VALUES (?,4,100,8,0,8,8,16,'file.py:1 fn')", + (event_id,), + ) + # Artificial boundary metrics exceed the real peak; they are not real events. + conn.execute("INSERT INTO trace_entry_0 VALUES (-1,2,100,16,0,999,999,999,'boundary')") + conn.executemany( + "INSERT INTO block_0 VALUES (?,?,?,?,?,?,?)", + [ + (0, 100, 8, 8, 1, 0, -1), + (-1, 200, 8, 4, 1, -1, -1), # Static/preexisting, unknown lifetime. + ], + ) + return root + + +def mutate_db(root, sql): + with closing(sqlite3.connect(root / "device_0/slice_00001.db")) as conn, conn: + conn.executescript(sql) + + +def test_valid_external_dataset_boundary_and_unknown_extensions(tmp_path): + root = make_dataset(tmp_path / "snapshot.pkl.msinsight") + data = manifest_data() + data["extensions"] = {"future.vendor": {"version": 999, "frames": "claimed"}} + (root / "manifest.json").write_text(json.dumps(data), encoding="utf-8") + mutate_db( + root, + "CREATE TABLE pt_snap_future (opaque TEXT); INSERT INTO pt_snap_future VALUES ('ignored');", + ) + before = {p: p.read_bytes() for p in root.rglob("*") if p.is_file()} + result = validate_dataset(root) + assert result.real_event_count == 4 + assert result.boundary_event_count == 2 + assert result.text_callstacks is True + assert result.structured_frames is False + assert result.query_execution is False + assert before == {p: p.read_bytes() for p in before} + assert not list(root.rglob("*-journal")) + context = Context(root / "device_0/slice_00000.db") + try: + assert context.callstack_layout == "v1" + finally: + context.close() + + +def test_address_reuse_is_not_identity_and_state_is_slice_local(tmp_path): + root = make_dataset(tmp_path / "dataset") + mutate_db( + root, + "INSERT INTO block_0 VALUES (2,100,8,8,1,2,-1); " + "INSERT INTO block_0 VALUES (-2,200,8,4,1,-1,-1); " + "UPDATE block_0 SET state=0 WHERE id=0;", + ) + assert validate_dataset(root).real_event_count == 4 + + +@pytest.mark.parametrize("ids", [[1, 2], [0, 2], [0, 0], [0, -1], [], [True], [0, 1.0]]) +def test_producer_rejects_nonzero_sparse_empty_or_invalid_real_ids(ids): + with pytest.raises(DatasetContractError): + validate_event_ids(ids) + + +def test_real_ids_use_actual_positions_not_maximum(): + assert validate_event_ids(iter([0, 1, 2])) == 3 + + +@pytest.mark.parametrize( + "field,value,code", + [ + ("schemaVersion", 2, "version"), + ("schemaVersion", True, "type"), + ("schemaVersion", 2**31, "range"), + ("eventsPerSlice", 0, "range"), + ("eventsPerSlice", 2**63, "range"), + ("eventsPerSlice", 2.0, "type"), + ("sourceFile", "", "range"), + ("cacheHash", None, "type"), + ("status", "failed", "status"), + ("devices", {}, "empty"), + ("devices", [], "type"), + ], +) +def test_manifest_top_level_errors(field, value, code): + data = manifest_data() + data[field] = value + with pytest.raises(DatasetContractError) as exc: + parse_manifest(data) + assert exc.value.code == code + assert exc.value.location == field + + +@pytest.mark.parametrize( + "field,value", + [ + ("eventCount", 0), + ("eventCount", 5), + ("eventCount", True), + ("sliceCount", 1), + ("sliceCount", 0), + ("readySlices", [0, 0]), + ("readySlices", [0]), + ("readySlices", [0, 2]), + ("readySlices", [True, 1]), + ], +) +def test_device_counts_readiness_and_empty_device_rejected(field, value): + data = manifest_data() + data["devices"]["0"][field] = value + with pytest.raises(DatasetContractError, match="devices.0"): + parse_manifest(data) + + +@pytest.mark.parametrize("device", ["01", "-1", "device_0", "1/../0", str(2**31)]) +def test_invalid_device_identity(device): + data = manifest_data() + data["devices"][device] = data["devices"].pop("0") + with pytest.raises(DatasetContractError): + parse_manifest(data) + + +@pytest.mark.parametrize( + "field,value", + [ + ("startEventId", 1), + ("startEventId", 3), + ("endEventId", 4), + ("endEventId", 1), + ("index", 0), + ("index", True), + ("ready", "true"), + ("file", "../other.db"), + ("file", "/other.db"), + ("file", "C:\\other.db"), + ("file", "device_1/slice_00001.db"), + ("file", "device_0/slice_00000.db"), + ("file", "device_0/./slice_00001.db"), + ("file", "device_0/slice_00001.db\0"), + ], +) +def test_slice_overlap_gaps_capacity_paths_and_types(field, value): + data = manifest_data() + data["devices"]["0"]["slices"][1][field] = value + with pytest.raises(DatasetContractError, match=r"devices.0.slices\[1\]"): + parse_manifest(data) + + +def test_building_inspection_is_distinct_from_complete_analysis(): + data = manifest_data() + data["status"] = "building" + data["devices"]["0"]["readySlices"] = [1] + data["devices"]["0"]["slices"][0]["ready"] = False + inspection = parse_manifest(data, require_complete=False) + with pytest.raises(DatasetContractError, match="building"): + inspection.require_complete() + with pytest.raises(DatasetContractError, match="building"): + parse_manifest(data) + data["status"] = "complete" + with pytest.raises(DatasetContractError, match="not_ready"): + parse_manifest(data, require_complete=False) + + +def test_opaque_empty_hash_is_not_a_verified_content_digest(): + data = manifest_data() + data["cacheHash"] = "" + assert parse_manifest(data).cache_hash == "" + + +def test_multiple_devices_and_short_slices_are_legal(): + data = manifest_data() + data["devices"]["7"] = deepcopy(data["devices"]["0"]) + for index, item in enumerate(data["devices"]["7"]["slices"]): + item["file"] = f"device_7/slice_{index:05d}.db" + data["eventsPerSlice"] = 3 + assert len(parse_manifest(data).devices) == 2 + + +@pytest.mark.parametrize( + "sql,code", + [ + ("DELETE FROM trace_entry_0 WHERE id=2;", "event_identity"), + ("UPDATE trace_entry_0 SET id=4 WHERE id=3;", "event_identity"), + ("UPDATE trace_entry_0 SET action=8 WHERE id=2;", "event_action"), + ("UPDATE trace_entry_0 SET action=4 WHERE id=-1;", "event_action"), + ("UPDATE trace_entry_0 SET action=NULL WHERE id=2;", "event_action"), + ("UPDATE trace_entry_0 SET active='bad' WHERE id=2;", "event_values"), + ("UPDATE trace_entry_0 SET reserved=-1 WHERE id=2;", "event_values"), + ("UPDATE trace_entry_0 SET callstack=x'00' WHERE id=2;", "event_values"), + ("CREATE TABLE CALLSTACK (id INTEGER, callstack TEXT);", "layout"), + ("CREATE TABLE TRACE_ENTRY_1 (id INTEGER);", "device_tables"), + ( + "ALTER TABLE block_0 RENAME TO saved; " + "CREATE TABLE block_0 (id INTEGER, address INTEGER, size INTEGER, " + "requestedSize INTEGER, state INTEGER, allocEventId INTEGER, freeEventId INTEGER, " + "PRIMARY KEY (id,address)); INSERT INTO block_0 SELECT * FROM saved;", + "primary_key", + ), + ("UPDATE dictionary SET \"value\"='wrong' WHERE \"column\"='state';", "dictionary"), + ("CREATE TABLE trace_entry_1 (id INTEGER);", "device_tables"), + ("ALTER TABLE block_0 RENAME TO block_1;", "device_tables"), + ("CREATE TABLE callstack (id INTEGER, callstack TEXT);", "layout"), + ("ALTER TABLE trace_entry_0 RENAME COLUMN callstack TO callstackId;", "columns"), + ("ALTER TABLE trace_entry_0 ADD COLUMN extra INTEGER;", "columns"), + ( + "ALTER TABLE trace_entry_0 RENAME TO saved; CREATE VIEW trace_entry_0 AS SELECT * FROM saved;", + "table", + ), + ("UPDATE block_0 SET id=7 WHERE id=0;", "block_identity"), + ("UPDATE block_0 SET allocEventId=-2 WHERE id=-1;", "block_identity"), + ("UPDATE block_0 SET freeEventId=-2 WHERE id=0;", "lifecycle"), + ("INSERT INTO block_0 VALUES (2,300,8,8,1,2,1);", "lifecycle"), + ("UPDATE block_0 SET address=300 WHERE id=0;", "block_identity"), + ("UPDATE block_0 SET freeEventId=3 WHERE id=0;", "block_identity"), + ("UPDATE block_0 SET requestedSize=9 WHERE id=0;", "block"), + ("UPDATE block_0 SET state=99 WHERE id=0;", "block"), + ], +) +def test_invalid_sqlite_artifacts_are_locatable(tmp_path, sql, code): + root = make_dataset(tmp_path / "dataset") + mutate_db(root, sql) + with pytest.raises(DatasetContractError) as exc: + validate_dataset(root) + assert exc.value.code == code + assert exc.value.location.startswith("device_0/slice_00001.db") + + +def test_missing_slice_does_not_create_database(tmp_path): + root = make_dataset(tmp_path / "dataset") + path = root / "device_0/slice_00001.db" + path.unlink() + with pytest.raises(DatasetContractError, match="missing"): + validate_dataset(root) + assert not path.exists() + + +def test_symlink_artifact_member_rejected(tmp_path): + root = make_dataset(tmp_path / "dataset") + path = root / "device_0/slice_00001.db" + path.unlink() + path.symlink_to(root / "device_0/slice_00000.db") + with pytest.raises(DatasetContractError, match="symlink"): + validate_dataset(root) + + +def test_invalid_json_duplicate_members_and_sqlite_fail_closed(tmp_path): + root = make_dataset(tmp_path / "dataset") + path = root / "manifest.json" + original = path.read_text(encoding="utf-8") + for content in ( + "{", + original.replace('"schemaVersion": 1', '"schemaVersion": 1, "schemaVersion": 1'), + ): + path.write_text(content, encoding="utf-8") + with pytest.raises(DatasetContractError): + validate_dataset(root) + path.write_text(original, encoding="utf-8") + (root / "device_0/slice_00001.db").write_bytes(b"not a database") + with pytest.raises(DatasetContractError, match="sqlite"): + validate_dataset(root) + + +@pytest.mark.parametrize( + "scope", + [ + QueryScope("dataset"), + QueryScope("device", 0), + QueryScope("slice", 0, 1), + QueryScope("event_range", 0, None, 1, 3), + ], +) +def test_scope_contract_not_execution(scope): + scope.validate(parse_manifest(manifest_data())) + + +@pytest.mark.parametrize( + "scope", + [ + QueryScope("dataset", 0), + QueryScope("device", 1), + QueryScope("device", 0, 1), + QueryScope("slice", 0, 2), + QueryScope("event_range", 0, None, -1, 3), + QueryScope("event_range", 0, None, 3, 1), + QueryScope("event_range", 0, None, 0, 4), + ], +) +def test_invalid_scope(scope): + with pytest.raises(DatasetContractError): + scope.validate(parse_manifest(manifest_data())) + + +def test_pinned_msinsight_enum_facts(): + # Fixed-revision snapshot_db.py, not native v2's additional oom=8 enum. + assert ACTION_NAMES == ( + "segment_map", + "segment_unmap", + "segment_alloc", + "segment_free", + "alloc", + "free_requested", + "free_completed", + "workspace_snapshot", + ) + assert BLOCK_STATES == ((-1, "inactive"), (0, "active_pending_free"), (1, "active_allocated")) + + +def test_bilingual_manifest_examples_are_identical_and_valid(): + root = Path(__file__).resolve().parents[1] + examples = [] + for language in ("en", "zh"): + text = (root / f"docs/{language}/sharded-snapshotdb.md").read_text(encoding="utf-8") + example = text.split("```json\n", 1)[1].split("```", 1)[0] + data = json.loads(example) + assert parse_manifest(data).devices[0].event_count == 4 + examples.append(data) + assert examples[0] == examples[1] + + +def test_missing_manifest_and_symlink_directory_fail_closed(tmp_path): + with pytest.raises(DatasetContractError, match="missing"): + validate_dataset(tmp_path / "absent") + root = make_dataset(tmp_path / "dataset") + alias = tmp_path / "alias" + alias.symlink_to(root, target_is_directory=True) + with pytest.raises(DatasetContractError, match="path"): + validate_dataset(alias) + + +@pytest.mark.parametrize("value", [None, [], "manifest"]) +def test_invalid_manifest_object(value): + with pytest.raises(DatasetContractError, match="type"): + parse_manifest(value) From 9e17e734a205eb2658f97475353379e2f7af4a02 Mon Sep 17 00:00:00 2001 From: codeasier Date: Mon, 5 Oct 2026 10:16:14 +0800 Subject: [PATCH 2/2] fix(docs): preserve dataset source evidence within branding contract (#198) --- docs/en/sharded-snapshotdb.md | 4 ++-- docs/zh/sharded-snapshotdb.md | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/en/sharded-snapshotdb.md b/docs/en/sharded-snapshotdb.md index 5c45d1d..6af500b 100644 --- a/docs/en/sharded-snapshotdb.md +++ b/docs/en/sharded-snapshotdb.md @@ -185,8 +185,8 @@ assert not inspection.structured_frames Interoperability facts were read from fixed upstream [manifest validation](https://github.com/Ascend/msinsight/blob/101f65b877a267ffd5f66ea3834706057ba243e5/server/src/modules/memsnapshot/service/MemSnapshotSliceService.cpp), [positional SQLite reading](https://github.com/Ascend/msinsight/blob/101f65b877a267ffd5f66ea3834706057ba243e5/server/src/modules/memsnapshot/database/MemSnapshotDatabase.cpp), -[base schemas](https://github.com/Ascend/msinsight/blob/101f65b877a267ffd5f66ea3834706057ba243e5/scripts/MemSnapDump/tools/adaptors/database/snapshot_db.py), -and [publication/backfills](https://github.com/Ascend/msinsight/blob/101f65b877a267ffd5f66ea3834706057ba243e5/scripts/MemSnapDump/tools/adaptors/snapshot2db.py). +[base schemas](https://api.github.com/repos/Ascend/msinsight/git/blobs/1fe50a5136f85365db55f4632d58f3cf7090431e), +and [publication/backfills](https://api.github.com/repos/Ascend/msinsight/git/blobs/95612509ef990ac6b7034919dae4938cc6acf3ad). Those source files carry Huawei's Mulan PSL v2 notice. This contract, Python validator and synthetic tests are original local implementation of interface facts; no upstream implementation is copied into the package. The upstream docs diff --git a/docs/zh/sharded-snapshotdb.md b/docs/zh/sharded-snapshotdb.md index 36652da..a07ce11 100644 --- a/docs/zh/sharded-snapshotdb.md +++ b/docs/zh/sharded-snapshotdb.md @@ -153,8 +153,8 @@ assert not inspection.structured_frames 接口事实已从固定版本真实源码读取: [manifest 校验](https://github.com/Ascend/msinsight/blob/101f65b877a267ffd5f66ea3834706057ba243e5/server/src/modules/memsnapshot/service/MemSnapshotSliceService.cpp)、 [SQLite 按位置读取](https://github.com/Ascend/msinsight/blob/101f65b877a267ffd5f66ea3834706057ba243e5/server/src/modules/memsnapshot/database/MemSnapshotDatabase.cpp)、 -[基础 schema](https://github.com/Ascend/msinsight/blob/101f65b877a267ffd5f66ea3834706057ba243e5/scripts/MemSnapDump/tools/adaptors/database/snapshot_db.py)、 -[发布/回填](https://github.com/Ascend/msinsight/blob/101f65b877a267ffd5f66ea3834706057ba243e5/scripts/MemSnapDump/tools/adaptors/snapshot2db.py)。 +[基础 schema](https://api.github.com/repos/Ascend/msinsight/git/blobs/1fe50a5136f85365db55f4632d58f3cf7090431e)、 +[发布/回填](https://api.github.com/repos/Ascend/msinsight/git/blobs/95612509ef990ac6b7034919dae4938cc6acf3ad)。 这些源码包含 Huawei Mulan PSL v2 声明。本文合同、Python 校验器和合成测试是本地原创的 接口事实实现,没有向包内复制上游实现;上游 docs 单独使用 CC BY 4.0,不能替代源码声明。