Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,11 @@ Nonmatching recognized targets require `--force`; unknown targets are preserved.
Publication/focus failures are compensated, with explicit recovery evidence if
rollback fails (force is not a single crash-atomic swap). See the
[native import guide](docs/en/quickstart.md#optional-import-a-native-sharded-dataset).
Only bounded dataset `event` queries are currently supported.
Dataset `event` addressing and point-event `active_blocks_at_event` /
`active_memory_callstack_at_event` share batched cross-shard alloc/free sources,
including original artifacts without extensions. Dataset-global peak/list/leak
queries and `report peak-memory` remain explicitly unsupported. See
[dataset attribution and coverage](docs/en/querying.md#dataset-point-event-attribution).

Explicit `pt-snap import snapshot.pkl --format msinsight --json` produces
`snapshot.pkl.msinsight/` for the fixed msinsight revision's **same original
Expand Down
5 changes: 4 additions & 1 deletion README_zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,10 @@ pt-snap split snapshot.pkl --slices 4 --output snapshot-slices
默认单库导入不变;复用校验全部片及内容/选项/语义版本。不匹配的已识别目标须
`--force`,未知目标保留。发布/focus 失败会补偿,回滚自身失败则保留并报告恢复证据
(force 不是单次崩溃原子 swap)。详见
[原生分库导入](docs/zh/quickstart.md#可选导入原生分库数据集);当前仅支持有界数据集 `event` 查询。
[原生分库导入](docs/zh/quickstart.md#可选导入原生分库数据集)。数据集 `event` 寻址及定点
`active_blocks_at_event` / `active_memory_callstack_at_event` 共用跨片 alloc/free 批量来源,
原版无扩展产物也适用。数据集全局 peak/list/leak 及 `report peak-memory` 仍明确不支持。
详见[数据集定点归因与覆盖](docs/zh/querying.md#数据集定点事件归因)。

显式 `pt-snap import snapshot.pkl --format msinsight --json` 生成
`snapshot.pkl.msinsight/`,对齐固定 msinsight 版本的**同一原始 pickle + 邻接缓存**入口。
Expand Down
85 changes: 85 additions & 0 deletions docs/en/querying.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,91 @@ group identity. Missing/static/preexisting groups have no captured allocation
stack to retrieve; omit `stack_bytes` on the original query to see their full
display labels. Results are read-only and summaries do not change the database.

## Dataset point-event attribution

A complete compatibility-v1 or native-v2 directory/manifest supports `event`
addressing plus `active_blocks_at_event` and `active_memory_callstack_at_event`
with a real `event_id`. The latter two route one device's actual event and use
its shard's complete lifecycle observations **after** that forward event:
`allocEventId <= E` and completed `freeEventId > E` (or `-1`). `free_requested`
does not remove active memory. Negative boundary rows represent the instant
before a shard's first event and are not query endpoints. Native sparse IDs are
not renumbered: `event(id=gap)` may return no rows inside an addressed slice;
active attribution at a nonexistent ID fails instead of inventing event 0.

```bash
pt-snap query '<dataset_dir>' --device 0 --template-use active_memory_callstack_at_event --params '{"event_id":700,"top_n":20}' --json
pt-snap query '<dataset_dir>' --device 0 --template-use active_blocks_at_event --params '{"event_id":700,"limit":10}' --exact-total --json
```

Replace the path/device/event with actual values. Foreign alloc/free events and
full v1 inline or v2 local-ID stacks are resolved **before** grouping/top_n, in
batches of at most 256 distinct event IDs per owning shard. No foreign real rows
are inserted. Original artifacts need neither metadata nor reference tables.
Lifetimes use dataset generation/device/alloc-event or stable negative identity,
not addresses; `state_scope=slice_observation` warns that stored `state` is not
necessarily the pending-free state at E. Block rows expose `allocation_source`,
`free_source`, their statuses and `lifecycle_id`. Missing dynamic sources retain
bytes and dynamic category, separate from static/preexisting unknown history.
`free=-1` means live **or unknown**, not a proven leak.

Dataset groups always expose full `source_stack_id`/`stack_id`, `stack_kind` and
`stack_event_id`. Default no-extension `event` rows keep their original nine
values; source/frame row fields appear with `stack_bytes>=0` or a recognized
ordered-frame declaration. `scope.source_coverage` is still supplied. Both dataset
layouts use full-text canonical identity; v2 looks up the owning slice's actual
local ID, returned as source `local_stack_id`/`slice_index` provenance, not a
cross-shard grouping key. Covered ordered arrays use only exact raw-frame identity,
independent of formatted text availability; `text_kind` reports that availability
separately. Explicitly covered arrays (including empty arrays) are captured raw
evidence. Groups prefer an available captured-text representative for retrieval. Nonempty whitespace and a captured literal `[missing callstack]`
are not missing. Never merge by local IDs or shortened display labels. `stack_bytes`
shortens display only; use `event(id=stack_event_id)` on the same dataset/device
for full text. Ordered raw frames, when explicitly covered, also participate in
identity; no text parsing reconstructs frames.

`scope.source_coverage` is independent of `has_more`/`truncated`/exact row totals:
active-query coverage describes **all filtered blocks before top_n/paging**, with
active/dynamic bytes, resolved allocation/free counts, captured dynamic bytes,
unknown/preexisting counts, ordered-frame coverage and degradation. For `event`,
its coverage scope is only `returned_events`. A complete manifest/range does not
prove complete historical sources or frames. Percentages still use included
bytes **after dynamic top_n but before max_rows**, not the dataset active counter.
Exact totals ignore row/rank caps, but do not change that percentage denominator.

One call shares its deadline across source batches, grouping and exact totals.
Manifest validation/hash and Context setup elapsed time are charged, but synchronous
filesystem I/O/validation is checked at phase boundaries, not hard-preempted; no
OS-level time guarantee is claimed. SQL uses cleared-on-exit progress handlers.
The Context LRU remains bounded (default four) and borrowed caches remain caller-owned.
No merged temporary database is created. `ReportService.event_attribution(E, ...)`
reuses this same query path for a supplied event. Dataset-global peak selection,
`report peak-memory`, cross-slice event ranges, arbitrary SQL/global list/leak/group
merging remain explicitly unsupported until their own contracts are implemented.

### Optional ordered-frame reader contract

This reader-only pt-snap extension is new, not an existing original-producer
feature. Current native/compatible writers emit **no** ordered-frame tables.
Declare `extensions.ptSnapOrderedFrames={"version":1}` in the manifest and use
isolated actual tables in each covered source shard:

- `pt_snap_frame_coverage(eventId INTEGER PRIMARY KEY, frameCount INTEGER)`
- `pt_snap_frame(eventId INTEGER, frameIndex INTEGER, frameJson TEXT, PRIMARY KEY(eventId,frameIndex))`

Raw objects must be finite standard JSON, with nesting depth at most 128; deeper
or nonfinite/unparseable input degrades safely, including recursion failures.
Each covered real event needs nonnegative integer `frameCount`, exactly indices
`0..frameCount-1`, and JSON **objects** preserving all raw typed fields. Order and
repeated frames are retained; empty arrays require an explicit count zero row.
Only recognized version/schema and valid per-event coverage produce `frames`
with `frames_status=ordered`. Unknown version, missing schema/rows, invalid JSON
or incomplete coverage degrade to `frames=null`, `frames_status=text_only`;
formatted text is never split/reversed into raw frames. P0/overview's dataset-wide
`structured_frames=false` remains a conservative validation capability, not a
claim about the scoped source payload. Tree rendering and GUI acceptance are not
provided by this extension interface.

## Query Templates

Templates are organized into three categories. Use `pt-snap capabilities --json` for the full catalog (CLI version, every template contract, and bundled skills), or `pt-snap query --list` to see names and descriptions. Filter with `--category`.
Expand Down
9 changes: 6 additions & 3 deletions docs/en/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,8 +96,10 @@ itself fails, recovery paths are preserved and explicitly reported. Do not delet
them before inspecting the evidence. Arbitrary I/O failures, concurrent writers
and crashes do not have unconditional atomic restoration guarantees.

Dataset focus currently supports only bounded `event` addressing; it does not
silently run peak/lifecycle/aggregate queries over one shard. See the
Dataset focus supports bounded `event` addressing and point-event active block/
callstack attribution with shared cross-shard sources. Dataset-global peak/list/leak
aggregates remain explicit errors, not one-shard approximations. See
[point-event coverage](querying.md#dataset-point-event-attribution) and the
[native manifest contract](sharded-snapshotdb.md#published-native-dataset-p2).
`SnapshotAnalyzer` remains analysis-only, without import/split methods.

Expand All @@ -123,7 +125,8 @@ Base tables are physical inline-v1 tables; metadata has import format `1`.
Callstack text is copied once **per event**, not once per distinct stack, so disk
and temporary conversion space can grow substantially. Native interning remains
unchanged. No ordered structured-frame capability is invented from formatted text.
Only bounded `event` dataset queries are currently available.
Bounded `event` and point-event active block/callstack dataset queries are
available; global aggregate/report selection remains unsupported.

Compatible publication is **no-replace**: external/original/user caches are never
overwritten or deleted, even under `--force`. A complete, attested, identical
Expand Down
22 changes: 15 additions & 7 deletions docs/en/sharded-snapshotdb.md
Original file line number Diff line number Diff line change
Expand Up @@ -170,7 +170,8 @@ which P0 does not implement. Text stacks cannot substitute for these guarantees.
There is no global event order across devices. Boundary IDs cannot be range
endpoints. P0 `DatasetValidation.query_execution=False` describes validation,
not execution. P1 accepts complete manifests/directories through the resolver
below; cross-slice execution remains deferred.
below; general cross-slice aggregation remains deferred; point-event source
resolution is described in the [querying guide](querying.md#dataset-point-event-attribution).

```python
from pt_snap_cli.core.dataset_contract import QueryScope, validate_dataset
Expand Down Expand Up @@ -218,16 +219,22 @@ with SnapshotAnalyzer(Path("/capture/snapshot.pkl.msinsight"), device_id=0) as a
print(event["scope"], page["total_is_exact"])
```

Only the `event` template is currently supported through dataset focus: a real ID
The `event` template supports bounded addressing through dataset focus: a real ID
routes to its containing slice; an inclusive `min_id/max_id` range must fit one
slice; `--slice`/`slice_index` lists that slice's real events. Negative IDs are not
whole-dataset selectors. Output `scope` identifies the actual DB/device/slice,
restricted real interval, dataset fingerprint and boundary exclusion. Totals and
pagination apply to this scope, **not** to an implicitly truncated full dataset.
Caller parameter defaults in CLI `effective_params` are additionally constrained
by `scope`. Cross-slice ranges, unbounded multi-slice queries and aggregate or
lifecycle templates fail explicitly; they never silently choose the first/latest
slice. Direct single-DB v1/v2 queries retain their existing semantics.
by `scope`. Point-event `active_blocks_at_event` and
`active_memory_callstack_at_event` now resolve alloc/free/full stack sources in
bounded batches before grouping/ranking, including original no-extension artifacts.
See [point-event attribution](querying.md#dataset-point-event-attribution) for
lifecycle identity, separate source/frame coverage, and the optional versioned
ordered-frame reader interface. Dataset-global aggregates, peak/list/leak queries,
cross-slice event ranges and unbounded multi-slice event queries still fail
explicitly; they never silently choose the first/latest slice. Direct single-DB
v1/v2 queries retain their existing semantics.

`core.dataset_resolver.DatasetResolver.inspect(path)` returns an immutable
`ResolvedDataset` (or `None` for a standalone DB). Its `paths(QueryScope(...))`
Expand Down Expand Up @@ -302,8 +309,9 @@ alias/sidecar/persistent-WAL checks and bounded borrowed-cache ownership. Focus,
metadata, overview and single-slice `event` addressing work without the source
pickle. Native metadata is available dataset-wide; external compatibility metadata
remains unavailable unless the recognized compatible attestation below is present.
Native mode implies no migration, repair, cross-slice aggregation/lifecycle execution,
compatible export, salted hash or GUI acceptance. P0 schema/manifest semantics are
Native mode implies no migration, repair, dataset-global aggregation/peak/list/leak,
compatible export, salted hash or GUI acceptance. Point-event source resolution is
available through the shared query path described above. P0 schema/manifest semantics are
unchanged; the compatibility-only finalized-WAL transport is described above.

Publication and force/focus compensation are described in the
Expand Down
66 changes: 66 additions & 0 deletions docs/zh/querying.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,72 @@ pt-snap query '<db_path>' --device 0 --template-use event --params '{"id":123}'
分组没有可取回的采集分配栈;在原查询中省略 `stack_bytes` 可查看完整显示标签。
查询保持只读,摘要不会改动数据库。

## 数据集定点事件归因

完整兼容 v1 或原生 v2 目录/manifest 支持 `event` 寻址,以及接受真实 `event_id` 的
`active_blocks_at_event`、`active_memory_callstack_at_event`。后两者按设备路由实际事件,
从所在片完整生命周期观察计算正向事件**之后**的集合:`allocEventId <= E` 且完成释放
`freeEventId > E`(或 `-1`)。`free_requested` 不移除 active 内存。负边界行表示片内
首事件之前,不是定点端点。原生稀疏 ID 不改号:片内 `event(id=gap)` 可返回空行,
不存在事件的 active 归因则明确失败,不虚构 event 0。

```bash
pt-snap query '<dataset_dir>' --device 0 --template-use active_memory_callstack_at_event --params '{"event_id":700,"top_n":20}' --json
pt-snap query '<dataset_dir>' --device 0 --template-use active_blocks_at_event --params '{"event_id":700,"limit":10}' --exact-total --json
```

请替换实际路径/设备/事件。片外 alloc/free 事件及完整 v1 内联或 v2 局部 ID 栈在
分组/top_n **之前**解析,按实际来源片每批至多 256 个不同事件 ID。不补入外片真实行。
原版产物无需 metadata 或引用表。生命周期用数据集 generation/设备/分配事件或稳定
负身份,不用地址;`state_scope=slice_observation` 提醒存储 `state` 不一定是 E 时的
pending-free 状态。block 行提供 `allocation_source`、`free_source`、状态和
`lifecycle_id`。缺失动态来源仍保留字节及动态类别,与 static/preexisting 未知历史分开;
`free=-1` 表示存活**或未知**,不证明泄漏。

数据集分组始终提供完整 `source_stack_id`/`stack_id`、`stack_kind`、`stack_event_id`。
无扩展的默认 `event` 行保留原始九个值;`stack_bytes>=0` 或已识别有序 frame 声明时
才增加来源/frame 行字段,`scope.source_coverage` 仍返回。
两种数据集布局均按完整文本的 canonical 身份分组;v2 仍按来源片真实局部 ID 查栈,
来源 `local_stack_id`/`slice_index` 是 provenance,不是跨片分组 key。有覆盖的有序数组
只使用精确原始 frame 身份,与格式化文本是否存在无关;`text_kind` 独立描述文本
可用性。明确覆盖的数组(含空数组)是已采集原始证据;分组优先选择有文本的代表
事件以便取回全文。非空 whitespace 和真实采集的字面量
`[missing callstack]` 不是 missing。禁止按局部 ID 或缩略显示标签合并。`stack_bytes`
只缩短显示;在同一数据集/设备用 `event(id=stack_event_id)` 取回全文。明确覆盖的原始
有序 frames 也参与身份;不从文本重建 frames。

`scope.source_coverage` 独立于 `has_more`/`truncated`/精确行数:active 查询覆盖的是
**top_n/分页之前全部筛选 block**,包括 active/dynamic 字节、已解析 alloc/free 数、采集
动态字节、未知/预存数、有序 frame 覆盖及降级。`event` 覆盖仅为 `returned_events`。
完整 manifest/范围不证明历史来源或 frame 完整。百分比分母仍为动态 top_n **之后、
max_rows 之前**所含字节,不是数据集 active 计数器。精确总数忽略行数/排名上限,但
不改变百分比分母。

一次调用的来源批次、分组和精确总数共用 deadline。manifest 校验/hash 及 Context
设置耗时计入预算,但同步文件 I/O/校验只在阶段边界检查,不硬抢占,不保证 OS 级
时间上限。SQL progress handler 退出即清除。Context LRU 有界(默认四个),借用
cache 仍由调用者所有。不生成合并临时数据库。`ReportService.event_attribution(E, ...)`
对给定事件复用同一路径。数据集全局峰值选择、`report peak-memory`、跨片事件范围、
任意 SQL/全局 list/leak/group 合并仍明确不支持,等待各自合同实现。

### 可选有序 frame reader 合同

这是新增的仅 reader pt-snap 扩展,不是原版 producer 已有功能。当前原生/兼容 writer
**不生成**有序 frame 表。manifest 声明 `extensions.ptSnapOrderedFrames={"version":1}`,
并在有覆盖的来源片使用隔离的实际表:

- `pt_snap_frame_coverage(eventId INTEGER PRIMARY KEY, frameCount INTEGER)`
- `pt_snap_frame(eventId INTEGER, frameIndex INTEGER, frameJson TEXT, PRIMARY KEY(eventId,frameIndex))`

原始 object 必须是有限数字的标准 JSON,嵌套深度至多 128;超深、非有限数或解析
失败(包括递归错误)安全降级。每个覆盖的真实事件须有非负整数 `frameCount`、恰好 `0..frameCount-1` 索引及保留全部
原始类型字段的 JSON **object**。保持顺序和重复帧;空数组须有明确 count=0 行。
只有已识别版本/schema 及有效逐事件覆盖才输出 `frames`、`frames_status=ordered`。
未知版本、缺失 schema/行、无效 JSON 或覆盖不全降级为 `frames=null`、
`frames_status=text_only`;不会切分/反转格式化文本伪造原始 frames。P0/overview 的
数据集级 `structured_frames=false` 仍是保守校验能力,不表示定点来源载荷必然无 frames。
本接口不提供树渲染或 GUI 验收。

## 查询模板

模板分为三个分类。用 `pt-snap capabilities --json` 一次查看完整清单(CLI 版本、全部模板契约和随包 skill),或用 `pt-snap query --list` 查看名称与描述。用 `--category` 过滤。
Expand Down
Loading