Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,9 +66,11 @@ rollback fails (force is not a single crash-atomic swap). See the
[native import guide](docs/en/quickstart.md#optional-import-a-native-sharded-dataset).
Dataset `event` addressing and point-event `active_blocks_at_event` /
`active_memory_callstack_at_event` share batched cross-shard alloc/free sources,
including original artifacts without extensions. Dataset-global peak/list/leak
queries and `report peak-memory` remain explicitly unsupported. See
[dataset attribution and coverage](docs/en/querying.md#dataset-point-event-attribution).
including original artifacts without extensions. Built-in global peaks, event pages,
canonical stack statistics, deduplicated lifecycles, terminal leak candidates and
`report peak-memory` share bounded core semantics; custom SQL/overrides remain unsupported
on datasets. See the [global support matrix](docs/en/querying.md#dataset-global-built-in-support-p3)
and [dataset attribution and coverage](docs/en/querying.md#dataset-point-event-attribution).

Explicit `pt-snap import snapshot.pkl --format msinsight --json` produces
`snapshot.pkl.msinsight/` for the fixed msinsight revision's **same original
Expand Down
6 changes: 4 additions & 2 deletions README_zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,8 +58,10 @@ pt-snap split snapshot.pkl --slices 4 --output snapshot-slices
(force 不是单次崩溃原子 swap)。详见
[原生分库导入](docs/zh/quickstart.md#可选导入原生分库数据集)。数据集 `event` 寻址及定点
`active_blocks_at_event` / `active_memory_callstack_at_event` 共用跨片 alloc/free 批量来源,
原版无扩展产物也适用。数据集全局 peak/list/leak 及 `report peak-memory` 仍明确不支持。
详见[数据集定点归因与覆盖](docs/zh/querying.md#数据集定点事件归因)。
原版无扩展产物也适用。内建全局峰值、事件分页、canonical 栈统计、去重生命周期、
终片泄漏候选及 `report peak-memory` 共用有界 core 语义;数据集仍不支持自定义 SQL/override。
详见[全局支持矩阵](docs/zh/querying.md#数据集全局内建支持p3)及
[数据集定点归因与覆盖](docs/zh/querying.md#数据集定点事件归因)。

显式 `pt-snap import snapshot.pkl --format msinsight --json` 生成
`snapshot.pkl.msinsight/`,对齐固定 msinsight 版本的**同一原始 pickle + 邻接缓存**入口。
Expand Down
97 changes: 88 additions & 9 deletions docs/en/querying.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ pt-snap query [DB_PATH] [--template-use <template_name>] [--params <json>] \
| `--category` | Filter templates by category: `basic`, `statistical`, `business` |
| `--template-info` | Show template details (parameters, output schema, and field semantics) |
| `-n` | Maximum displayed rows; zero or a negative value means unlimited. Separate from `--timeout`. |
| `--exact-total` | Run a `COUNT` of the matching set. Default `total` is the returned-row count. |
| `--exact-total` | Count the complete matching set: standalone SQL COUNT or dataset post-merge count. Default `total` is returned rows. |
| `--timeout` | Wall-clock budget for one `query` / `QueryService` call (page query and optional `--exact-total` COUNT share it) via a SQLite progress handler. `<= 0` disables. Default: `PT_SNAP_QUERY_TIMEOUT` or unlimited. The environment variable applies to every template query, including `report peak-memory`. |
| `--json` | Emit machine-readable JSON (execute, `--list`, and `--template-info`) |

Expand Down Expand Up @@ -78,6 +78,76 @@ group identity. Missing/static/preexisting groups have no captured allocation
stack to retrieve; omit `stack_bytes` on the original query to see their full
display labels. Results are read-only and summaries do not change the database.

## Dataset-global built-in support (P3)

Complete native-v2 and compatibility-v1 datasets share the following **one-device**
query support matrix, also exposed as each catalog/template-info `dataset_support`.
Device selection stays explicit → focused → first discovered; peaks are never summed
across devices and there is no shared cross-device event order.

| Template | Dataset semantics before the final output window |
| --- | --- |
| `event`, `allocation` | Filter all selected real events, globally sort with the declared ID tie-break, then offset/limit/max_rows; negative boundaries excluded |
| `memory_peak` | Independent allocated/active/reserved maxima over real events; earliest real ID wins ties; `start_id/end_id` may cross shards |
| `allocator_gap` | The same independently selected global events, with each gap computed from counters **at that event** |
| `active_blocks_at_event`, `active_memory_callstack_at_event` | Route a real event to its owner and resolve full cross-shard sources before grouping/top_n (details below) |
| `preexisting_live` | Owner-event preexisting occupancy, not a leak count |
| `block` | Merge proved allocation-event lifetimes and stable negative tokens once, then apply filters/order/page; latest containing-shard state is an observation, not state at arbitrary E |
| `freed_block_lifetime` | Deduplicated captured allocations with a proved `free_completed` source; event-ID distance buckets, not time |
| `leak_detection` | Captured dynamic lifetimes still present without completion in the **terminal dataset shard**, not union of per-shard unfreed lists; survival is only a candidate, not a confirmed leak; `--slice` is rejected |
| `callstack_analysis` | All real stack-bearing trace actions, including free/segment/workspace, merged by full canonical source identity before global thresholds/ranking; weighted `avg_size=total_size/alloc_count` |
| Runtime overrides/custom templates/arbitrary SQL | Explicitly unsupported on datasets, even when named after a built-in; pass an explicit standalone member to use local SQL |

`memory_peak` and `allocator_gap` semantics version **2** exclude negative synthetic
rows in **standalone** SQL too. `callstack_analysis` version **2** retains the legacy
`alloc_count` column name but does **not** mean allocation count: v1 counts all real
non-NULL-text rows; v2 counts all real non-NULL stack references, retaining absent/NULL
joined text as one distinct missing group. Empty text counts as its own group, separate
from missing and captured literal labels. Both standalone layouts now group full text,
not local v2 IDs. Datasets also accept explicitly covered ordered
raw arrays as stack evidence; missing text does not change raw-array identity. Size
sums include nonallocation actions and are not instantaneous live occupancy.

The shared YAML output schemas declare every optional dataset-only row column;
standalone row shapes stay unchanged. `block` and dataset `leak_detection` expose
`lifecycle_id`, proof/source statuses, source payloads, latest observation slice/state
scope, category and terminal survival; leak rows additionally retain `requestedSize`,
`state` and `freeEventId`. These observations and proofs are not leak confirmation.
Dataset statistics expose `source_stack_id`, `stack_kind`, `text_kind`,
`stack_event_id` and `frames_status`. The representative real event prefers nonempty
captured text when available but is not the canonical group identity and need not be
an allocation action; otherwise it may have empty/NULL text. `text_kind` is provenance
only and never changes raw-array identity. Retrieve actual text/arrays using `event`
on that same dataset/device; `frames_status` is not frame reconstruction.

```bash
pt-snap query '<dataset_dir>' --device 0 --template-use memory_peak --params '{"start_id":100,"end_id":700}' --json
pt-snap query '<dataset_dir>' --device 0 --template-use event --params '{"min_id":100,"max_id":700,"limit":20,"offset":20}' --exact-total --json
pt-snap query '<dataset_dir>' --device 0 --template-use leak_detection -n 20 --exact-total --json
pt-snap report peak-memory '<dataset_dir>' --device 0 --metric reserved --start-id 100 --end-id 700 --timeout 10 --json
```

Use actual bounds. Ranges may include sparse gaps; they never create an event.
An empty real range has NULL peak values/IDs. Outside/negative/reversed range
endpoints fail; `event(id=gap)` may return no rows, while active-at-gap fails.
Global filters/dedup/source identity/thresholds/sort precede the final window.
`exact_total` ignores row caps; default total is returned rows with honest exactness
flags. Result `has_more/truncated` describes the window, not source/range completeness.
`scope` records requested bounds, actual real-event coverage where applicable,
involved `slice_indices`, device, fingerprint and boundary exclusion. Source coverage
is separate. Unproved historical lifecycle identities are shard-local, explicitly
marked, never guessed by address/local stack ID and never asserted as proved leaks.

Dataset reads detach batches before foreign-owner lookups, using the borrowed bounded
Context LRU. One diminishing deadline includes all shards, sources, grouping and exact
totals; reports explicitly share **one** budget across their three queries. Cumulative
fetched/output work is limited to **100000 rows / 64 MiB serialized values**, including
repeated source reads. Exceeding either fails the entire operation, never an alleged
complete partial result. Materialization/sort are bounded by this work ceiling, **not**
by max_rows. This is not a process RSS limit: Python/SQLite, one batch and individual
cells have overhead. Manifest hashing/validation remains proportional to artifact size
and phase-checked, not hard-preempted. No temporary merge DB, repair or artifact write.

## Dataset point-event attribution

A complete compatibility-v1 or native-v2 directory/manifest supports `event`
Expand Down Expand Up @@ -136,9 +206,9 @@ filesystem I/O/validation is checked at phase boundaries, not hard-preempted; no
OS-level time guarantee is claimed. SQL uses cleared-on-exit progress handlers.
The Context LRU remains bounded (default four) and borrowed caches remain caller-owned.
No merged temporary database is created. `ReportService.event_attribution(E, ...)`
reuses this same query path for a supplied event. Dataset-global peak selection,
`report peak-memory`, cross-slice event ranges, arbitrary SQL/global list/leak/group
merging remain explicitly unsupported until their own contracts are implemented.
reuses this same query path for a supplied event. Dataset-global queries and reports
follow the explicit support matrix above; arbitrary SQL/custom overrides never inherit
those merge semantics merely by using a built-in name.

### Optional ordered-frame reader contract

Expand Down Expand Up @@ -358,7 +428,7 @@ This is useful because `reserved` may peak at a different event from `active` or
For a higher-level summary, use the report command:

```bash
pt-snap report peak-memory [db_path] [--device <id>] [--metric active|allocated|reserved] [--include-static|--exclude-static] [--limit <n>] [--json]
pt-snap report peak-memory [db_path] [--device <id>] [--metric active|allocated|reserved] [--include-static|--exclude-static] [--limit <n>] [--start-id <id>] [--end-id <id>] [--timeout <seconds>] [--json]
```

Examples:
Expand All @@ -380,7 +450,12 @@ The report command combines:
- `allocator_gap`
- `active_memory_callstack_at_event`

and prints either a human-readable summary or JSON.
and prints either a human-readable summary or JSON. `--start-id/--end-id` restrict
peak selection; attribution still resolves the complete active set at that selected
event, including earlier allocations. `--timeout` (or `PT_SNAP_QUERY_TIMEOUT`) is
one diminishing **report-wide** deadline, not three reset per-query timeouts.
JSON adds peak `scope`, separate attribution `source_coverage`, `timeout_s`, and
`budget_scope="report_composition"`.

The existing `callstack_groups` and `percent_of_active_blocks` values are preserved.
JSON also exposes attribution `has_more`, `truncated`, `total_is_exact`, and
Expand Down Expand Up @@ -417,8 +492,10 @@ An empty trace returns no groups, zero included bytes, and unknown coverage.
`-n` still caps `rows` and `returned`. By default `total` equals `returned`
(this page's row count) and `total_is_exact` is true only when the page is
the complete matching set (`has_more` is false and `offset` is 0).
`--exact-total` runs a `COUNT` of the matching set (ignoring `limit` /
`offset` / `top_n`) and sets `total_is_exact` to true. When a finite
`--exact-total` counts the matching set (ignoring `limit` / `offset` / `top_n`)
and sets `total_is_exact` to true. Standalone queries use SQL `COUNT`; dataset
built-ins count the fully merged/deduplicated matching set before its global window,
not capped shard-local rows. On the standalone path, when a finite
trailing `LIMIT` is in effect, the executor fetches one extra row to set
`has_more` without that count. A finite inner `top_n` (for example inside
a CTE) is not a trailing `LIMIT`: when that ranked window is full,
Expand All @@ -434,7 +511,9 @@ change the row cap. A non-numeric `PT_SNAP_QUERY_TIMEOUT` is
`INVALID_PARAMETER`, not `QUERY_FAILED`.

`effective_params` is the validated parameter set after defaults and `choices`
normalization. When `-n` is set, `limit` is the trailing SQL LIMIT after the
normalization. Dataset `limit` reports the final global output-window cap (with the
same min-of-positive-caps rule), not a per-shard SQL LIMIT; global filters and merges
are already complete. On the standalone path, when `-n` is set, `limit` is the trailing SQL LIMIT after the
same merge the executor applies: `min` of a declared template `limit` and
`-n`, or `-n` appended when the rendered SQL has no trailing LIMIT. Inner
caps such as `top_n` stay their own parameters; they do not rewrite
Expand Down
35 changes: 19 additions & 16 deletions docs/en/sharded-snapshotdb.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,11 @@
## Scope and versions

This defines artifact contracts, shared validators and an **explicit** compatible
CLI exporter below, not a cross-slice query engine. Existing `import` defaults to one
CLI exporter below. The [P3 built-in query matrix](querying.md#dataset-global-built-in-support-p3)
defines supported cross-slice execution separately. Existing `import` defaults to one
native v2 DB; `split` still produces replayable pickle/JSON. Existing single-DB
v1/v2 single-DB analysis is unchanged. The P1 reader below additionally accepts a
v1/v2 analysis retains its entry points; P3 real-event peak/gap and all-action full-text
stack-statistics semantics are explicitly versioned in the querying guide. The P1 reader below additionally accepts a
complete compatibility-v1 manifest or directory as focus. The `complete` gate
applies only to datasets, not to legacy single-DB analysis.

Expand Down Expand Up @@ -170,8 +172,8 @@ which P0 does not implement. Text stacks cannot substitute for these guarantees.
There is no global event order across devices. Boundary IDs cannot be range
endpoints. P0 `DatasetValidation.query_execution=False` describes validation,
not execution. P1 accepts complete manifests/directories through the resolver
below; general cross-slice aggregation remains deferred; point-event source
resolution is described in the [querying guide](querying.md#dataset-point-event-attribution).
below; the explicit [P3 support matrix](querying.md#dataset-global-built-in-support-p3)
and [point-event sources](querying.md#dataset-point-event-attribution) define execution.

```python
from pt_snap_cli.core.dataset_contract import QueryScope, validate_dataset
Expand All @@ -198,8 +200,9 @@ pt-snap query --template-use event --slice 1 --json
`overview.dataset` reports format, manifest version/status, content fingerprint,
real/boundary counts, per-device slice paths/ranges and capabilities. Event bounds
exclude negative synthetic rows (also for standalone DB overview). Text stacks
are available, but not necessarily populated; structured frames and cross-slice
queries are **unavailable**, including when unknown extensions claim them.
are available, but not necessarily populated; P0 structured-frame validation remains
**unavailable** despite unknown extension claims. Cross-slice built-ins follow the P3
matrix; recognized ordered-frame query coverage is separate from P0 validation.
External dataset-wide import metadata is unavailable; per-slice metadata alone
is not a source attestation. The compatible exporter below supplies a recognized,
whole-artifact `ptSnap` attestation, making its validated metadata available.
Expand All @@ -220,8 +223,8 @@ with SnapshotAnalyzer(Path("/capture/snapshot.pkl.msinsight"), device_id=0) as a
```

The `event` template supports bounded addressing through dataset focus: a real ID
routes to its containing slice; an inclusive `min_id/max_id` range must fit one
slice; `--slice`/`slice_index` lists that slice's real events. Negative IDs are not
routes to its containing slice; inclusive `min_id/max_id` ranges may cross shards
with global filtering/sorting/paging; `--slice`/`slice_index` restricts real events. Negative IDs are not
whole-dataset selectors. Output `scope` identifies the actual DB/device/slice,
restricted real interval, dataset fingerprint and boundary exclusion. Totals and
pagination apply to this scope, **not** to an implicitly truncated full dataset.
Expand All @@ -231,10 +234,10 @@ by `scope`. Point-event `active_blocks_at_event` and
bounded batches before grouping/ranking, including original no-extension artifacts.
See [point-event attribution](querying.md#dataset-point-event-attribution) for
lifecycle identity, separate source/frame coverage, and the optional versioned
ordered-frame reader interface. Dataset-global aggregates, peak/list/leak queries,
cross-slice event ranges and unbounded multi-slice event queries still fail
explicitly; they never silently choose the first/latest slice. Direct single-DB
v1/v2 queries retain their existing semantics.
ordered-frame reader interface. Dataset-global built-ins use the explicit P3 matrix
and cumulative work ceilings; arbitrary SQL/custom overrides fail explicitly, never
silently choosing the first/latest slice. Single-DB real-event peak/gap and all-action
full-text stack-statistics versions are documented with the same contracts.

`core.dataset_resolver.DatasetResolver.inspect(path)` returns an immutable
`ResolvedDataset` (or `None` for a standalone DB). Its `paths(QueryScope(...))`
Expand Down Expand Up @@ -262,7 +265,7 @@ This is read-only validation, not a filesystem sandbox or a producer lock.
Native-v2 private staging has **no published manifest**, so it is not a dataset;
individual DBs remain readable via the single-DB entry. Published native format
and compatible export are separate explicit protocols below. Cross-slice lifecycle
execution remains separate work; safe original-WAL reading is not GUI acceptance.
execution follows the P3 matrix; safe original-WAL reading is not GUI acceptance.

## Published Native Dataset (P2)

Expand Down Expand Up @@ -309,8 +312,8 @@ alias/sidecar/persistent-WAL checks and bounded borrowed-cache ownership. Focus,
metadata, overview and single-slice `event` addressing work without the source
pickle. Native metadata is available dataset-wide; external compatibility metadata
remains unavailable unless the recognized compatible attestation below is present.
Native mode implies no migration, repair, dataset-global aggregation/peak/list/leak,
compatible export, salted hash or GUI acceptance. Point-event source resolution is
Native mode implies no migration, repair, compatible export, salted hash or GUI
acceptance. Dataset-global built-ins follow the P3 query matrix. Point-event source resolution is
available through the shared query path described above. P0 schema/manifest semantics are
unchanged; the compatibility-only finalized-WAL transport is described above.

Expand Down Expand Up @@ -386,7 +389,7 @@ JOIN gaps. Unsupported OOM/unknown actions, nonzero/sparse IDs and empty/static-
selections fail; native retains its wider contract. All selected members finalize,
close and validate before complete publication. `pt_snap_block_reference` v1 is an
isolated text extension, not ordered frames or a claim of cross-source queries.
Those queries and global aggregation remain separate work. Text cannot recover
Those queries and global aggregation follow the P3 query matrix. Text cannot recover
original ordered frame arrays; original workspace raw-frame loss and string-action
OOM coercion are known upstream limitations, not compatible features to imitate.

Expand Down
Loading