Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,16 @@ Complete compatibility-v1 sharded artifacts can be focused by directory or
`manifest.json`, without the archived pickle. See [dataset focus and addressing](docs/en/sharded-snapshotdb.md#complete-dataset-focus-and-addressing-p1)
for read-only overview, event routing, `--slice`, and explicit cross-slice limits.

Opt into a **native, not msinsight-compatible** dataset with
`pt-snap import snapshot.pkl --events-per-slice 50000 --output-dir captures --json`.
The output is `captures/snapshot.pkl.pt-snap-native-v2/`; default standalone import
is unchanged. Reuse validates all shards and content/options/semantic versions.
Nonmatching recognized targets require `--force`; unknown targets are preserved.
Publication/focus failures are compensated, with explicit recovery evidence if
rollback fails (force is not a single crash-atomic swap). See the
[native import guide](docs/en/quickstart.md#optional-import-a-native-sharded-dataset).
Only bounded dataset `event` queries are currently supported.

## Commands

| Command | Description |
Expand Down Expand Up @@ -102,7 +112,7 @@ See the [documentation index](docs/README.md) for all English and Chinese guides
| Splitting snapshots | [Splitting Snapshots](docs/en/splitting.md) |
| Agent skills | [Agent Skills](docs/en/skills.md) |
| Database format | [SnapshotDB Schema](docs/en/database.md) |
| Sharded contract and internal replay (P0/P1; no CLI exporter) | [Sharded SnapshotDB protocol](docs/en/sharded-snapshotdb.md) |
| Sharded protocols, native import and compatibility limits | [Sharded SnapshotDB protocol](docs/en/sharded-snapshotdb.md) |
| Python API | [SnapshotAnalyzer API](docs/en/snapshot-analyzer-api.md) |
| Result mapping utility | [ResultMapper API](docs/en/result-mapper-api.md) |

Expand Down
9 changes: 8 additions & 1 deletion README_zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,13 @@ pt-snap split snapshot.pkl --slices 4 --output snapshot-slices
只读 overview、事件路由、`--slice` 和跨片限制见
[完整数据集 focus 与寻址](docs/zh/sharded-snapshotdb.md#完整数据集-focus-与寻址p1)。

用 `pt-snap import snapshot.pkl --events-per-slice 50000 --output-dir captures --json`
显式生成**原生、非 msinsight 兼容**数据集 `captures/snapshot.pkl.pt-snap-native-v2/`。
默认单库导入不变;复用校验全部片及内容/选项/语义版本。不匹配的已识别目标须
`--force`,未知目标保留。发布/focus 失败会补偿,回滚自身失败则保留并报告恢复证据
(force 不是单次崩溃原子 swap)。详见
[原生分库导入](docs/zh/quickstart.md#可选导入原生分库数据集);当前仅支持有界数据集 `event` 查询。

## 命令

| 命令 | 说明 |
Expand Down Expand Up @@ -95,7 +102,7 @@ helper 会先路由采集,无需先安装 CLI。分析仍需要 CLI 和 Snapsh
| 拆分快照 | [拆分快照](docs/zh/splitting.md) |
| Agent skill | [Agent Skills](docs/zh/skills.md) |
| 数据库格式 | [SnapshotDB Schema](docs/zh/database.md) |
| 分片合同与内部回放(P0/P1,无 CLI 导出器) | [分片 SnapshotDB 协议](docs/zh/sharded-snapshotdb.md) |
| 分片协议、原生导入与兼容边界 | [分片 SnapshotDB 协议](docs/zh/sharded-snapshotdb.md) |
| Python API | [SnapshotAnalyzer API](docs/zh/snapshot-analyzer-api.md) |
| 结果映射工具 | [ResultMapper API](docs/zh/result-mapper-api.md) |

Expand Down
20 changes: 20 additions & 0 deletions docs/en/database.md
Original file line number Diff line number Diff line change
Expand Up @@ -386,3 +386,23 @@ CREATE TABLE pt_snap_metadata (

Legacy and externally produced compatible databases may omit this table. They
remain queryable, but `pt-snap metadata` reports their metadata as unavailable.

### Published native datasets

`import --events-per-slice N` publishes a distinct `pt-snap-native-v2` manifest,
not a compatibility-v1 manifest or another standalone DB layout. Each finalized
shard uses the existing v2 tables plus `pt_snap_block_reference` (version 1), with
`blockId INTEGER PRIMARY KEY`, `stream INTEGER NOT NULL`, `allocCallstack TEXT`,
`freeCallstack TEXT`. These are text lifecycle sources, not structured frames or
an automatic cross-shard JOIN. `pt_snap_metadata` is required and identical in all
members; native manifest identity additionally versions source/device/capacity,
format/manifest/import/extension contracts, independently of CLI release version.

Real counts use positions per device, preserving original sparse IDs and native
OOM/workspace events. Negative boundaries do not consume capacity. Reader
validation includes full-member hashes and finalized identity/reference consistency
before bounded `event` addressing; it never repairs or migrates files. Aggregate
and lifecycle dataset queries remain unsupported. See the separate
[native manifest protocol](sharded-snapshotdb.md#published-native-dataset-p2) and
[transaction/recovery guide](quickstart.md#optional-import-a-native-sharded-dataset).
P0 compatibility-v1 table order, inline callstack layout and reject rules are unchanged.
27 changes: 27 additions & 0 deletions docs/en/focus-management.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,33 @@ ignore rules automatically.
}
```

## Complete Dataset Focus and Import Compensation

A complete compatibility-v1 dataset or published `pt-snap-native-v2` directory/
`manifest.json` can be focused with the same precedence and device rules:

```bash
pt-snap focus captures/snapshot.pkl.pt-snap-native-v2 --device 0 --json
pt-snap query --template-use event --slice 0 --json
```

Native layout is `v2`; compatibility layout is `v1`. All declared members are
validated before writing focus, with no artifact repairs or layout persistence.
Native sparse IDs are addressed as original IDs, not positional counts. Only
bounded `event` queries are supported through dataset focus; aggregate/lifecycle
and cross-slice requests fail explicitly. API session focus is read-only and
`SnapshotAnalyzer` has no import/split facade.

Import writes project focus in the **current directory**, not the output directory.
`--no-focus` skips it. A native dataset import focus failure, including mutation before an error,
restores the old artifact plus exact prior focus bytes and in-memory config state
(or prior absence) on ordinary compensation. If restoration fails, the error
identifies the retained `.focus.pt-snap-*.recovery` checkpoint and/or dataset
recovery directory. An empty checkpoint with `prior focus existed: False` records
prior absence, not a valid focus JSON. Preserve evidence for manual recovery;
no unconditional I/O/crash atomicity or concurrent-writer lock is promised.
See [native import](quickstart.md#optional-import-a-native-sharded-dataset).

## Error Handling

**No focus configured:**
Expand Down
52 changes: 52 additions & 0 deletions docs/en/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,58 @@ copying raw events directly. The resulting SnapshotDB records event-by-event
the supported input for `pt-snap query` and `pt-snap report`. This replay behavior
is part of the command workflow; snapshot runtime Python modules are not a public API.

### Optional: Import a Native Sharded Dataset

```bash
pt-snap import snapshot.pkl --events-per-slice 50000 --output-dir captures --json
pt-snap overview captures/snapshot.pkl.pt-snap-native-v2 --json
pt-snap query --template-use event --slice 0 --json
```

Without `--events-per-slice`, import still produces the existing single DB. With
it, the output is `<output-dir>/<full-input-filename>.pt-snap-native-v2/`, or the
same directory beside the source if `--output-dir` is omitted. `--format
pt-snap-native-v2` is optional in this mode; `--format single-db` forbids a capacity.
`--format compatibility-v1` and `msinsight` are reserved and explicitly rejected:
native output is **not** an msinsight-compatible export.

Capacity counts real chronological positions **per device**, not `max(id)+1` or
negative synthetic boundaries. Original sparse/nonzero IDs, OOM and workspace
events remain native. `--device` selects one event-bearing device; otherwise all
such devices are imported. Empty/static-only and unselected device positions are
reported in `omitted_devices`; selecting an empty device or an all-empty input
fails. Sharding does not bound pickle loading, registry/block rows or peak RSS.

Reuse requires a validated complete manifest and **every** closed member, with
matching source SHA256, device selection, capacity, native format, manifest,
semantic import, extension and metadata/layout contracts. CLI release version
changes alone do not rebuild. Native identity is separate from future compatible
format identities and from msinsight's salted `cacheHash`. JSON adds `dataset_path`,
`devices`, `slice_count`, `format`, `omitted_devices`, `reused` and `rebuilt`;
`db_path` denotes the dataset directory in this mode.

A matching target is reused. A different/corrupt recognized native artifact is
preserved unless `--force` explicitly requests replacement. Unknown directories,
extra unrelated files, malformed manifests and symlink targets are never adopted
or deleted, even with force: choose a new `--output-dir`. All devices/shards are
staged beside the target, finalized, closed, metadata-attested and validated; the
source is hashed again before publication. No partial shard is a reusable cache.

New publication uses an exclusive directory rename. Force replacement first
renames the old artifact into an owned recovery directory, then publishes the
new one; **there is a transient path gap, not a single atomic swap**. Requested
project focus is part of compensation, including a focus write that mutates
before raising: ordinary failure restores old artifact and exact old focus
bytes/state (or previous absence). `--no-focus` skips focus entirely. If rollback
itself fails, recovery paths are preserved and explicitly reported. Do not delete
them before inspecting the evidence. Arbitrary I/O failures, concurrent writers
and crashes do not have unconditional atomic restoration guarantees.

Dataset focus currently supports only bounded `event` addressing; it does not
silently run peak/lifecycle/aggregate queries over one shard. See the
[native manifest contract](sharded-snapshotdb.md#published-native-dataset-p2).
`SnapshotAnalyzer` remains analysis-only, without import/split methods.

### Optional: Split a Snapshot

Use `pt-snap split` when you need smaller, independently replayable files. Split
Expand Down
62 changes: 59 additions & 3 deletions docs/en/sharded-snapshotdb.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Sharded SnapshotDB protocol (P0, compatibility v1)
# Sharded SnapshotDB protocols (compatibility v1 and native v2)

[中文](../zh/sharded-snapshotdb.md) | English

Expand Down Expand Up @@ -247,8 +247,64 @@ mode; standalone DB behavior is unchanged. This is read-only inspection, not a
filesystem sandbox or a concurrent producer lock. Native-v2 replay's private staging directory has **no published manifest**
and is rejected as a dataset; its individual DBs remain readable via the existing
single-DB entry. A future compatibility-v1 producer can use this same reader
without producer-specific metadata. No native-v2 manifest protocol, exporter,
cache publication, migration, or cross-slice lifecycle reconstruction is invented.
without producer-specific metadata. The P1 reader does not create a native manifest; publication requires the explicit
P2 native protocol below. Compatibility export and cross-slice lifecycle execution
remain separate work. In particular, the fixed original compatibility producer can
leave finalized, sidecar-free WAL headers, which this reader still rejects. Transport
support for those artifacts is a compatibility prerequisite, not passed upstream/GUI
interoperability; analysis never rewrites them.

## Published Native Dataset (P2)

This is an **original, separately named and versioned** pt-snap protocol, not a
relabeled P0 artifact. `ImportService`/CLI can publish the internal native replay
result as `<output-dir>/<full-source-filename>.pt-snap-native-v2/`. The root must
contain exactly `manifest.json` plus the declared `device_<id>` directories; each
device directory contains exactly its canonical `slice_<index:05d>.db` members.

Required native manifest fields:

| Field | Contract |
| --- | --- |
| `format` | Exactly `pt-snap-native-v2`, independently dispatched from compatibility-v1 |
| `schemaVersion`, `status` | Native manifest version `1`, only `complete` |
| `sourceFile` | Nonempty source basename matching `metadata.source_name`; never opened by analysis |
| `identity` | Exact fields `sourceSha256`, `requestedDevice`, `eventsPerSlice`, `outputFormat`, `manifestContractVersion`, `formatContractVersion`, `importContractVersion`, `extensionContractVersion`, `metadataSchemaVersion`, `importFormatVersion` |
| `metadata` | Complete existing `ImportMetadata` object, identical to the single row in every shard |
| `omittedDevices` | Distinct unselected/empty/static-only source device positions, disjoint from selected devices |
| `devices` | Canonical decimal keys; positive `eventCount`, `sliceCount`, all `readySlices` in order, ordered `slices` |
| Each slice | `index`, `startPosition`, `endPosition`, `startEventId`, `endEventId`, canonical relative `file`, `ready: true`, closed-member `sha256` |

All semantic contract versions start at `1`; native metadata schema is `1` and DB
import format is `2`. `requestedDevice` is null (all event-bearing devices) or one
nonnegative integer. SHA256 strings are lowercase 64-digit hex. CLI
`metadata.importer_version` is informational, **not** a cache invalidation axis.
Native and future compatible identities are separately versioned; no native
`cacheHash` is generated or accepted as an identity substitute.

Positions continuously cover `0..eventCount-1` per device, with every nonfinal
slice full and the final slice at most capacity. ID ranges are inclusive original
chronological bounds and may be sparse/nonzero; real count comes from positions
and actual rows, never `max(id)+1`. Negative synthetic rows are counted separately.
Each member has exactly its native-v2 trace/block device tables, `dictionary`,
`callstack`, `pt_snap_metadata`, and finalized `pt_snap_block_reference` v1. Native
trace columns match the standalone v2 schema (`callstackId`, not inline text),
with native OOM action 8 retained. Block IDs/lifetimes and text references must be
finalized, consistent across slices and refer to original IDs in that device.
Unknown extension declarations/tables do not grant structured-frame capability.

The native resolver validates all member hashes, actual schemas/dictionaries,
real count/ranges, metadata and finalized references, retaining the P1 pre-open
alias/sidecar/persistent-WAL checks and bounded borrowed-cache ownership. Focus,
metadata, overview and single-slice `event` addressing work without the source
pickle. Native metadata is available dataset-wide; compatibility metadata remains
unavailable. No migration, repair, cross-slice aggregation/lifecycle execution,
msinsight exporter, salted hash policy or GUI acceptance is implied. The P0
schema and reject policies above remain unchanged.

Publication and force/focus compensation are described in the
[import guide](quickstart.md#optional-import-a-native-sharded-dataset). A complete
manifest becomes visible only with all finalized members, never one ready shard.

## Internal continuous replay (P1)

Expand Down
16 changes: 16 additions & 0 deletions docs/zh/database.md
Original file line number Diff line number Diff line change
Expand Up @@ -368,3 +368,19 @@ CREATE TABLE pt_snap_metadata (

旧版或外部生成的兼容数据库可以不包含此表,仍可正常查询;`pt-snap metadata` 会将其
metadata 状态报告为 unavailable。

### 已发布原生数据集

`import --events-per-slice N` 发布独立 `pt-snap-native-v2` manifest,不是兼容 v1
manifest,也不改变单库布局。各最终化片使用已有 v2 表及 `pt_snap_block_reference`
(版本 1):`blockId INTEGER PRIMARY KEY`、`stream INTEGER NOT NULL`、
`allocCallstack TEXT`、`freeCallstack TEXT`。它们记录文本生命周期来源,不是结构化
frames 或自动跨片 JOIN。每片必需且一致的 `pt_snap_metadata` 之外,原生 manifest
身份另包含源/设备/容量、格式/manifest/导入/扩展合同版本,与 CLI 发布版本分离。

真实计数按各设备位置,保留原始稀疏 ID、OOM/workspace;负边界不占容量。
reader 在有界 `event` 寻址前校验完整成员哈希、最终化身份/引用一致性,不修复、不迁移。
数据集聚合/生命周期查询仍不支持。详见独立
[原生 manifest 协议](sharded-snapshotdb.md#已发布原生数据集p2)及
[事务恢复指南](quickstart.md#可选导入原生分库数据集)。
P0 兼容 v1 表序、内联 callstack 布局和拒绝规则不变。
22 changes: 22 additions & 0 deletions docs/zh/focus-management.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,28 @@ pt-snap config --json # 同一组操作用机器可读 JSON 输出
}
```

## 完整数据集 focus 与导入补偿

完整兼容 v1 数据集或已发布 `pt-snap-native-v2` 目录/`manifest.json` 可以按相同的
优先级和设备规则 focus:

```bash
pt-snap focus captures/snapshot.pkl.pt-snap-native-v2 --device 0 --json
pt-snap query --template-use event --slice 0 --json
```

原生布局为 `v2`,兼容布局为 `v1`;写 focus 前校验全部声明成员,不修复产物,不保存
布局信息。原生稀疏 ID 按原始 ID 寻址,不当成位置计数。数据集 focus 只支持有界
`event`,聚合/生命周期及跨片请求明确失败。API 会话 focus 只读,`SnapshotAnalyzer`
不承担 import/split。

导入把项目 focus 写入**当前目录**,不是输出目录;`--no-focus` 跳过。原生数据集导入 focus 失败
(包括先修改再抛错)在普通补偿下恢复旧产物与旧 focus 原始字节及内存 config 状态,
或者恢复原先不存在。恢复失败时,错误指明保留的 `.focus.pt-snap-*.recovery` checkpoint
及/或数据集恢复目录。若 checkpoint 为空且 `prior focus existed: False`,它记录原先
不存在,不是有效 focus JSON。请保留证据供人工恢复;不承诺任意 I/O/崩溃原子性或并发
writer 锁。详见[原生导入](quickstart.md#可选导入原生分库数据集)。

## 错误处理

**未设置焦点时查询:**
Expand Down
Loading