Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,14 @@ rollback fails (force is not a single crash-atomic swap). See the
[native import guide](docs/en/quickstart.md#optional-import-a-native-sharded-dataset).
Only bounded dataset `event` queries are currently supported.

Explicit `pt-snap import snapshot.pkl --format msinsight --json` produces
`snapshot.pkl.msinsight/` for the fixed msinsight revision's **same original
pickle + adjacent cache** entrance. Inline stacks cost more space than native
interning. Existing compatible targets are never replaced, even with `--force`;
only fully validated identical pt-snap caches are reused. GUI acceptance remains
**pending/not run**; hash equality alone is not acceptance. See
[compatible export and limits](docs/en/sharded-snapshotdb.md#explicit-msinsight-compatible-export-p2).

## Commands

| Command | Description |
Expand Down
6 changes: 6 additions & 0 deletions README_zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,12 @@ pt-snap split snapshot.pkl --slices 4 --output snapshot-slices
(force 不是单次崩溃原子 swap)。详见
[原生分库导入](docs/zh/quickstart.md#可选导入原生分库数据集);当前仅支持有界数据集 `event` 查询。

显式 `pt-snap import snapshot.pkl --format msinsight --json` 生成
`snapshot.pkl.msinsight/`,对齐固定 msinsight 版本的**同一原始 pickle + 邻接缓存**入口。
内联栈比原生去重占更多空间;即使 `--force` 也不替换已有兼容目标,只复用完整校验且
身份相同的 pt-snap 缓存。GUI **待验收/未运行**,hash 相等不代表验收通过。详见
[兼容导出及限制](docs/zh/sharded-snapshotdb.md#显式-msinsight-兼容导出p2)。

## 命令

| 命令 | 说明 |
Expand Down
26 changes: 25 additions & 1 deletion docs/en/database.md
Original file line number Diff line number Diff line change
Expand Up @@ -405,4 +405,28 @@ before bounded `event` addressing; it never repairs or migrates files. Aggregate
and lifecycle dataset queries remain unsupported. See the separate
[native manifest protocol](sharded-snapshotdb.md#published-native-dataset-p2) and
[transaction/recovery guide](quickstart.md#optional-import-a-native-sharded-dataset).
P0 compatibility-v1 table order, inline callstack layout and reject rules are unchanged.
P0 compatibility-v1 table order, inline layout and manifest semantics are unchanged.

### Explicit compatible export

`import --format msinsight` (alias `compatibility-v1`) publishes physical v1
`trace_entry_<device>` tables with the ninth column `callstack TEXT`, not views,
with the exact positional block/dictionary schema. It removes staging's unprefixed
`callstack` table and writes `pt_snap_metadata.import_format_version=1`, retaining
isolated `pt_snap_block_reference` text lifecycle sources. Native standalone and
native datasets keep their interning and v2 metadata. Structured ordered frames
are not emitted or inferred from text.

Inline stack bytes scale with the sum of stack text **over every event**, not
unique stacks; conversion temporarily holds native and inline tables and may leave
SQLite free pages. Capacity does not bound blocks, loading RSS or total disk.
Compatible datasets additionally carry `ptSnap.identity`, metadata, omitted devices
and full member hashes. Recognized whole-artifact metadata is available only after
all attestation checks; external v1 datasets still need no metadata.

For finalized compatibility datasets ONLY, bare validation and cached analysis
use `mode=ro&immutable=1` after rejecting all live/dangling sidecars and aliases
before the first SQLite open. Closed/checkpointed original WAL bytes/inventory
remain unchanged. This assumes no active writer; native dataset non-WAL policy and
standalone `mode=ro` defaults are unchanged. See
[compatible protocol and no-replace rules](sharded-snapshotdb.md#explicit-msinsight-compatible-export-p2).
11 changes: 11 additions & 0 deletions docs/en/focus-management.md
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,17 @@ prior absence, not a valid focus JSON. Preserve evidence for manual recovery;
no unconditional I/O/crash atomicity or concurrent-writer lock is promised.
See [native import](quickstart.md#optional-import-a-native-sharded-dataset).

Explicit compatible import (`--format msinsight`) has the same requested-focus
compensation, but **never replaces** an existing compatible output, even with
force. Only fully attested identical pt-snap caches can be reused; otherwise use
a new output. GUI still requires the same original pickle plus adjacent cache,
although pt-snap focus needs only the complete artifact. All compatible members
are checked for aliases/live/dangling sidecars before any SQLite open; bare
validation and cached queries use `mode=ro&immutable=1` for finalized closed
members, preserving original checkpointed-WAL bytes/inventory. This is not a live
writer lock; native non-WAL checks and standalone defaults remain unchanged.


## Error Handling

**No focus configured:**
Expand Down
43 changes: 41 additions & 2 deletions docs/en/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,8 +61,8 @@ Without `--events-per-slice`, import still produces the existing single DB. With
it, the output is `<output-dir>/<full-input-filename>.pt-snap-native-v2/`, or the
same directory beside the source if `--output-dir` is omitted. `--format
pt-snap-native-v2` is optional in this mode; `--format single-db` forbids a capacity.
`--format compatibility-v1` and `msinsight` are reserved and explicitly rejected:
native output is **not** an msinsight-compatible export.
Native output is **not** an msinsight-compatible export. Explicit `--format
msinsight` (alias `compatibility-v1`) selects the separate compatible mode below.

Capacity counts real chronological positions **per device**, not `max(id)+1` or
negative synthetic boundaries. Original sparse/nonzero IDs, OOM and workspace
Expand Down Expand Up @@ -101,6 +101,45 @@ silently run peak/lifecycle/aggregate queries over one shard. See the
[native manifest contract](sharded-snapshotdb.md#published-native-dataset-p2).
`SnapshotAnalyzer` remains analysis-only, without import/split methods.

### Optional: Export an msinsight-Compatible Dataset

```bash
pt-snap import snapshot.pkl --format msinsight --json
pt-snap overview snapshot.pkl.msinsight --json
pt-snap query --template-use event --slice 0 --json
```

This **explicit** mode targets `Ascend/msinsight@101f65b877a267ffd5f66ea3834706057ba243e5`
and its original-pickle import entrance, not direct directory import. Keep the
same original pickle beside `<full-input-filename>.msinsight/` for GUI discovery;
pt-snap itself can analyze a relocated complete artifact without pickle. Optional
`--events-per-slice N` defaults to `500000`; `--output-dir` selects another parent,
not an independently supported GUI entrance. `--device` selects an event-bearing
device and omissions are reported. Nonzero/sparse real IDs, OOM/unknown actions,
empty/static-only selections fail, without silent remapping. Native mode still
preserves sparse IDs/OOM/workspace. Trusted pickle and full-input memory costs apply.

Base tables are physical inline-v1 tables; metadata has import format `1`.
Callstack text is copied once **per event**, not once per distinct stack, so disk
and temporary conversion space can grow substantially. Native interning remains
unchanged. No ordered structured-frame capability is invented from formatted text.
Only bounded `event` dataset queries are currently available.

Compatible publication is **no-replace**: external/original/user caches are never
overwritten or deleted, even under `--force`. A complete, attested, identical
pt-snap cache is reused (also with force); otherwise choose a new `--output-dir`.
Source content, sourceFile, options, all member hashes and contract versions must
match independently of salted `cacheHash`. A GUI-derived cache modification can
invalidate this attestation, without implying producer rerun. All staged members
close/finalize/validate before exclusive publication and requested focus; ordinary
failure compensates exact old focus bytes, with recovery evidence if rollback fails.
Standalone/native force policies remain unchanged. `SnapshotAnalyzer` is analysis-only.

**GUI acceptance is pending/not run.** Complete/ready, safe paths/device tables,
and derived allocation-cache availability/buildability also matter; matching hash
alone proves neither GUI reuse nor display parity. Details and fixed source evidence:
[compatible protocol](sharded-snapshotdb.md#explicit-msinsight-compatible-export-p2).

### Optional: Split a Snapshot

Use `pt-snap split` when you need smaller, independently replayable files. Split
Expand Down
118 changes: 94 additions & 24 deletions docs/en/sharded-snapshotdb.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@

## Scope and versions

This freezes an artifact contract and shared validation models, **not a new CLI
exporter or cross-slice query engine**. Existing `import` still defaults to one
This defines artifact contracts, shared validators and an **explicit** compatible
CLI exporter below, not a cross-slice query engine. Existing `import` defaults to one
native v2 DB; `split` still produces replayable pickle/JSON. Existing single-DB
v1/v2 single-DB analysis is unchanged. The P1 reader below additionally accepts a
complete compatibility-v1 manifest or directory as focus. The `complete` gate
Expand All @@ -21,7 +21,8 @@ Separate version axes:

- `schemaVersion=1`: manifest and compatibility base-table contract defined here.
- `pt_snap_metadata.import_format_version`: existing pt-snap importer format,
currently 2; **not** the manifest version and not required for external artifacts.
`2` for native and `1` for compatible inline export; **not** the manifest version
and not required for external artifacts.
- Future structured-frame extensions have their own version. Unknown extensions
do not invalidate base reading or imply structured-frame support.

Expand Down Expand Up @@ -198,8 +199,9 @@ real/boundary counts, per-device slice paths/ranges and capabilities. Event boun
exclude negative synthetic rows (also for standalone DB overview). Text stacks
are available, but not necessarily populated; structured frames and cross-slice
queries are **unavailable**, including when unknown extensions claim them.
Dataset-wide import metadata is unavailable; per-slice metadata is not a source
attestation for the entire dataset.
External dataset-wide import metadata is unavailable; per-slice metadata alone
is not a source attestation. The compatible exporter below supplies a recognized,
whole-artifact `ptSnap` attestation, making its validated metadata available.

Path precedence stays explicit → `PT_SNAP_DB_PATH` → nearest project focus → legacy
global config; device precedence stays explicit → focused → first discovered.
Expand Down Expand Up @@ -238,21 +240,22 @@ lookup even when size/mtime are unchanged. Analyzer-owned caches close on exit;
injected caches remain caller-owned. Validation opens/closes slices sequentially.

Building, missing/inconsistent slices, unknown base versions and unsafe paths
are rejected before focus writes or query execution. Before opening **any** slice,
the resolver checks every member's canonical non-symlink path, SQLite header and
WAL/journal/SHM sidecars (including dangling symlinks). Persistent WAL mode is
rejected even after checkpointing has removed its sidecars: `mode=ro` can recreate
them. The reader does not repair, checkpoint, delete sidecars or change journal
mode; standalone DB behavior is unchanged. This is read-only inspection, not a
filesystem sandbox or a concurrent producer lock. Native-v2 replay's private staging directory has **no published manifest**
and is rejected as a dataset; its individual DBs remain readable via the existing
single-DB entry. A future compatibility-v1 producer can use this same reader
without producer-specific metadata. The P1 reader does not create a native manifest; publication requires the explicit
P2 native protocol below. Compatibility export and cross-slice lifecycle execution
remain separate work. In particular, the fixed original compatibility producer can
leave finalized, sidecar-free WAL headers, which this reader still rejects. Transport
support for those artifacts is a compatibility prerequisite, not passed upstream/GUI
interoperability; analysis never rewrites them.
are rejected before focus writes or query execution. Before **any** SQLite open,
both the bare P0 validator and resolver check **all** member/parent aliases and
live/dangling WAL, journal and SHM sidecars. Finalized, closed compatibility-v1
members use `mode=ro&immutable=1`, including cached query Context connections.
This accepts the fixed original producer's checkpointed, sidecar-free WAL headers
without creating sidecars or changing bytes. The immutable flag assumes a closed
finalized artifact, **not a live-WAL shortcut** or concurrent-writer guarantee.
No repair, checkpoint, journal-mode change, migration or deletion occurs.
Native datasets retain persistent-WAL rejection; standalone Context defaults stay
`mode=ro` without immutable. Cache identity includes transport mode and generation.
This is read-only validation, not a filesystem sandbox or a producer lock.

Native-v2 private staging has **no published manifest**, so it is not a dataset;
individual DBs remain readable via the single-DB entry. Published native format
and compatible export are separate explicit protocols below. Cross-slice lifecycle
execution remains separate work; safe original-WAL reading is not GUI acceptance.

## Published Native Dataset (P2)

Expand Down Expand Up @@ -297,10 +300,11 @@ The native resolver validates all member hashes, actual schemas/dictionaries,
real count/ranges, metadata and finalized references, retaining the P1 pre-open
alias/sidecar/persistent-WAL checks and bounded borrowed-cache ownership. Focus,
metadata, overview and single-slice `event` addressing work without the source
pickle. Native metadata is available dataset-wide; compatibility metadata remains
unavailable. No migration, repair, cross-slice aggregation/lifecycle execution,
msinsight exporter, salted hash policy or GUI acceptance is implied. The P0
schema and reject policies above remain unchanged.
pickle. Native metadata is available dataset-wide; external compatibility metadata
remains unavailable unless the recognized compatible attestation below is present.
Native mode implies no migration, repair, cross-slice aggregation/lifecycle execution,
compatible export, salted hash or GUI acceptance. P0 schema/manifest semantics are
unchanged; the compatibility-only finalized-WAL transport is described above.

Publication and force/focus compensation are described in the
[import guide](quickstart.md#optional-import-a-native-sharded-dataset). A complete
Expand Down Expand Up @@ -353,6 +357,72 @@ rows, total staged bytes, registry size, or peak RSS. Synthetic regressions and
reviewed expandable/multi-device fixtures cover replay equivalence and resource
failure paths; no live upstream GUI or performance acceptance is claimed.

## Explicit msinsight-Compatible Export (P2)

`pt-snap import snapshot.pkl --format msinsight --json` (alias
`--format compatibility-v1`) targets **only**
`Ascend/msinsight@101f65b877a267ffd5f66ea3834706057ba243e5`. Its supported entrance is
opening the **same original pickle** in that target with adjacent
`<original-path>.msinsight/`, not direct standalone-directory GUI import.
`--output-dir` selects another parent for pt-snap; relocation alone does not create
an upstream GUI entrance. `sourceFile` is the absolute original source path.
Capacity defaults to `500000` and can be set with `--events-per-slice`; default
standalone import and explicit native-v2 output remain separate and unchanged.

The first-party converter consumes closed/finalized native staging, materializes
exact positional inline-v1 **tables**, removes unprefixed native `callstack`, and
writes physical-format metadata `1`. Original real IDs, actions, stream/metrics,
stable device-local block identities, `free_completed` and out-of-slice lifecycle
text references are preserved. No foreign real events are inserted to hide local
JOIN gaps. Unsupported OOM/unknown actions, nonzero/sparse IDs and empty/static-only
selections fail; native retains its wider contract. All selected members finalize,
close and validate before complete publication. `pt_snap_block_reference` v1 is an
isolated text extension, not ordered frames or a claim of cross-source queries.
Those queries and global aggregation remain separate work. Text cannot recover
original ordered frame arrays; original workspace raw-frame loss and string-action
OOM coercion are known upstream limitations, not compatible features to imitate.

`cacheHash = SHA256(b"mem_snapshot_parser_v2" + raw_pickle_bytes)`: the salt is
**prepended**. Independent `ptSnap.identity` fields are `sourceSha256`,
`requestedDevice`, `eventsPerSlice`, `outputFormat=compatibility-v1`,
`manifestContractVersion=1`, `exportContractVersion=1`, `referenceContractVersion=1`,
`metadataSchemaVersion=1`, `importFormatVersion=1`, and full `targetRevision`.
The `ptSnap` object also contains `metadata`, `omittedDevices`, and `members`
(canonical relative path → finalized SHA256). Whole validated ownership/identity
requires every member/hash and identical per-slice metadata; unknown/external
caches and unrelated files are not adopted. CLI version alone does not invalidate.

Compatible publication is **no-replace even with force**. Identical recognized
pt-snap caches reuse without loading pickle (force does not bypass this rule).
Changed source path/content, options/contracts, corrupted members or GUI-modified
DBs require a new `--output-dir`; no repair, overwrite or deletion of the old
artifact occurs. Exclusive publication and focus compensation use the existing
checked transaction; source SHA256 is rechecked immediately before publish/reuse.
Native force replacement and recovery semantics are untouched.

Inline text costs `sum(len(callstack.encode("utf-8")) for each event)` rather than
one copy per distinct stack. Existing native interning remains available. Conversion
temporarily holds both layouts and can leave free SQLite pages; capacity is not an
RSS/registry/block/disk bound. No optional frame enhancement is emitted.

**GUI remains pending/not run.** Fixed-source parser checks complete manifest,
nonempty identical hash, every ready/resolved/existing member, read-only DB/device
validation and allocation-cache existence **or buildability** before UP_TO_DATE.
Derived allocation caches may modify DBs without invoking the producer: mtime or
new derived tables alone are not rerun evidence. Later GUI acceptance must retain
UP_TO_DATE/no-script-invocation logs and compare curves, blocks and cross-slice
presentation. Source checks and original-artifact production reads are not GUI,
C++ server/compile or full differential/performance acceptance.

Immutable fixed blob evidence:
[parser predicates](https://api.github.com/repos/Ascend/msinsight/git/blobs/40046e6a92ff5af776ee55bd4cdd4586745af3c2),
[salt order](https://api.github.com/repos/Ascend/msinsight/git/blobs/ca421c7b33f3a0050df9e7a01e666f558801bc2f),
[allocation-cache predicates/build](https://api.github.com/repos/Ascend/msinsight/git/blobs/3d1ecc0ebd0d8e3eb0c984eda6b750ec4ce0e4f1).
The Ascend source is Mulan PSL v2, docs separately CC BY 4.0. This minimal converter
is original interface-facts code, not a copied vendor implementation or automatic
MIT relicensing of Ascend blobs. Existing pt-snap runtime lineage is a separate
license/provenance history.

## Evidence and limits

Interoperability facts were read from fixed upstream
Expand Down
Loading