Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 21 additions & 9 deletions docs/en/querying.md
Original file line number Diff line number Diff line change
Expand Up @@ -148,13 +148,18 @@ marked, never guessed by address/local stack ID and never asserted as proved lea

Dataset reads detach batches before foreign-owner lookups, using the borrowed bounded
Context LRU. One diminishing deadline includes all shards, sources, grouping and exact
totals; reports explicitly share **one** budget across their three queries. Cumulative
totals; reports explicitly share **one** budget across peak selection and attribution. Cumulative
fetched/output work is limited to **100000 rows / 64 MiB serialized values**, including
repeated source reads. Exceeding either fails the entire operation, never an alleged
complete partial result. Materialization/sort are bounded by this work ceiling, **not**
by max_rows. This is not a process RSS limit: Python/SQLite, one batch and individual
cells have overhead. Manifest hashing/validation remains proportional to artifact size
and phase-checked, not hard-preempted. No temporary merge DB, repair or artifact write.
cells have overhead. Strong manifest/member validation and hashing remain the default
and share that deadline: hash chunks, Python row batches and SQLite progress handlers
check cancellation. Individual filesystem calls are not OS-level hard-preempted.
`PT_SNAP_DATASET_CACHE=immutable` explicitly opts into process-local validation reuse
only when the caller guarantees immutable finalized files and reliable filesystem
change times; see [dataset validation and cache contracts](sharded-snapshotdb.md).
No temporary merge DB, repair or artifact write.

## Dataset point-event attribution

Expand Down Expand Up @@ -209,9 +214,10 @@ bytes **after dynamic top_n but before max_rows**, not the dataset active counte
Exact totals ignore row/rank caps, but do not change that percentage denominator.

One call shares its deadline across source batches, grouping and exact totals.
Manifest validation/hash and Context setup elapsed time are charged, but synchronous
filesystem I/O/validation is checked at phase boundaries, not hard-preempted; no
OS-level time guarantee is claimed. SQL uses cleared-on-exit progress handlers.
Manifest validation/hash and Context setup elapsed time are charged. Validation checks
cancellation between hash chunks and Python row batches, and within SQLite via
cleared-on-exit progress handlers. Individual filesystem calls are not hard-preempted;
no OS-level time guarantee is claimed.
The Context LRU remains bounded (default four) and borrowed caches remain caller-owned.
No merged temporary database is created. `ReportService.event_attribution(E, ...)`
reuses this same query path for a supplied event. Dataset-global queries and reports
Expand Down Expand Up @@ -454,17 +460,23 @@ pt-snap report peak-memory /path/to/snapshot.db --json

The report command combines:

- `memory_peak`
- `allocator_gap`
- `allocator_gap`, whose result also supplies the six existing `peak_*` fields
- `active_memory_callstack_at_event`

and prints either a human-readable summary or JSON. `--start-id/--end-id` restrict
peak selection; attribution still resolves the complete active set at that selected
event, including earlier allocations. `--timeout` (or `PT_SNAP_QUERY_TIMEOUT`) is
one diminishing **report-wide** deadline, not three reset per-query timeouts.
one diminishing **report-wide** deadline, never reset between queries.
JSON adds peak `scope`, separate attribution `source_coverage`, `timeout_s`, and
`budget_scope="report_composition"`.

A dataset report inspects the dataset once and reuses that validated fingerprint
for peak selection and attribution. Changes to the manifest or members detected
between or during these steps reject the whole report; this is a generation check,
not a concurrent-writer lock. Counter peaks are computed once for the selected
range. Attribution still consumes the same cumulative work budget, so a very large
active set can legitimately exceed that budget even when a peak-only query succeeds.

The existing `callstack_groups` and `percent_of_active_blocks` values are preserved.
JSON also exposes attribution `has_more`, `truncated`, `total_is_exact`, and
`effective_params` (`event_id`, `include_static`, `min_size`, `top_n`). A full
Expand Down
13 changes: 5 additions & 8 deletions docs/en/sharded-snapshotdb.md
Original file line number Diff line number Diff line change
Expand Up @@ -245,10 +245,8 @@ addresses all matching files even for a cross-slice range, but does not execute
By default **every independent inspection** fully validates the finalized contract
and hashes the manifest/members. File metadata alone is not content-integrity proof:
unknown or coarse filesystems may expose unchanged `ctime_ns` after a write.
Each query, including each query step within a multi-step report, calls `inspect`
again. Reports do not currently pin one `ResolvedDataset` across their steps or
invoke `ResolvedDataset.require_unchanged()`; there is no additional report-level
content-hash guard.
Pinned report-operation guards also rehash content by default, without repeating
row validation.

`PT_SNAP_DATASET_CACHE=immutable` explicitly opts into a process-wide LRU of at most
16 validated generations, without open connections. Set it only when you control
Expand All @@ -260,14 +258,13 @@ the tool. Unset it to retain strong validation; unknown values also use strong m
Windows and platforms without a usable POSIX change token remain in strong mode.

With that explicit opt-in, new resolver/service instances can reuse validated
results. Every cache hit during inspection still rechecks canonical paths,
results. Every hit and pinned-operation guard still rechecks canonical paths,
sidecars, native membership and each file's device, inode, size, mtime, **ctime**,
mode and link count. These conservative checks catch ordinary replacements and
in-place edits with restored mtime, but do not replace the immutable-file contract.
For a fixed shard count, an opted-in hit performs no row scans or whole-file hashes:
work is O(shards), independent of event/block counts. This metadata-only reuse also
applies to report query steps; a cache hit does not rehash content. Default inspections
and cache misses remain proportional to content. Explicit `validate_dataset` /
work is O(shards), independent of event/block counts. Default inspections and cache
misses remain proportional to content. Explicit `validate_dataset` /
`validate_native_dataset` calls always validate fully. Native admission verifies
hashes before and after validation and reuses the verified hashes for its fingerprint,
avoiding a third member-hash pass.
Expand Down
24 changes: 16 additions & 8 deletions docs/zh/querying.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,11 +128,14 @@ exactness 标记诚实。`has_more/truncated` 描述窗口,不表示来源/范
独立。未证明历史身份按片局部单独表达,不按地址/局部栈 ID 猜合并,不断言为已证明泄漏。

数据集读取先分离批次再查外片来源,使用借用的有界 Context LRU。所有片、来源、分组、
精确总数共用递减 deadline;报告三条查询明确共用**一个**预算。累计取回/输出工作上限为
精确总数共用递减 deadline;报告的峰值选择与归因明确共用**一个**预算。累计取回/输出工作上限为
**100000 行 / 64 MiB 序列化值**,含重复来源读取;超限整体失败,不返回冒充完整的部分
结论。物化/排序由工作上限约束,**不由 max_rows 约束**;不是进程 RSS 上限,Python/
SQLite、一个批次及单个 cell 有额外开销。manifest hash/校验仍与产物大小成正比且仅在
阶段边界检查,不硬抢占。不生成合并临时 DB、不修复或写产物。
SQLite、一个批次及单个 cell 有额外开销。默认仍对 manifest/成员执行完整校验和 hash,
并共用该 deadline:hash 分块、Python 行批次及 SQLite progress handler 都检查取消。
单个文件系统调用不提供 OS 级硬抢占。仅当调用方保证最终产物不可变且文件系统变更时间
可靠时,才可通过 `PT_SNAP_DATASET_CACHE=immutable` 显式启用进程内校验复用;参见
[数据集校验与缓存合同](sharded-snapshotdb.md)。不生成合并临时 DB、不修复或写产物。

## 数据集定点事件归因

Expand Down Expand Up @@ -176,8 +179,9 @@ max_rows 之前**所含字节,不是数据集 active 计数器。精确总数
不改变百分比分母。

一次调用的来源批次、分组和精确总数共用 deadline。manifest 校验/hash 及 Context
设置耗时计入预算,但同步文件 I/O/校验只在阶段边界检查,不硬抢占,不保证 OS 级
时间上限。SQL progress handler 退出即清除。Context LRU 有界(默认四个),借用
设置耗时计入预算。校验在 hash 分块与 Python 行批次之间检查取消,SQLite 通过退出即
清除的 progress handler 检查取消。单个文件系统调用不硬抢占,不保证 OS 级时间上限。
Context LRU 有界(默认四个),借用
cache 仍由调用者所有。不生成合并临时数据库。`ReportService.event_attribution(E, ...)`
对给定事件复用同一路径。数据集全局查询及报告遵循上文明确支持矩阵;任意 SQL/
自定义 override 不会因使用内建名字就继承全局合并语义。
Expand Down Expand Up @@ -397,15 +401,19 @@ pt-snap report peak-memory /path/to/snapshot.db --json

这个 report 命令会组合:

- `memory_peak`
- `allocator_gap`
- `allocator_gap`,其结果同时提供原有的六个 `peak_*` 字段
- `active_memory_callstack_at_event`

并输出人类可读摘要或 JSON。`--start-id/--end-id` 限制峰值选择;归因仍解析所选事件
完整活跃集合,包含更早分配。`--timeout`(或 `PT_SNAP_QUERY_TIMEOUT`)是递减的
**报告整体** deadline,不是重置三次的逐查询超时。JSON 新增峰值 `scope`、独立归因
**报告整体** deadline,不会在查询之间重置。JSON 新增峰值 `scope`、独立归因
`source_coverage`、`timeout_s` 与 `budget_scope="report_composition"`。

数据集报告只检查一次数据集,峰值选择与归因复用同一个已验证 fingerprint。
若在步骤之间或执行期间检测到 manifest 或成员变化,整个报告会失败;这是版本检查,
不是并发写入锁。所选区间只计算一次 counter 峰值。归因仍消耗同一个累计工作预算,
因此活跃集合很大时,即使单独峰值查询成功,报告仍可能合理地超出预算。

原有 `callstack_groups` 和 `percent_of_active_blocks` 数值保持不变。JSON 还会
返回归因的 `has_more`、`truncated`、`total_is_exact` 和 `effective_params`
(`event_id`、`include_static`、`min_size`、`top_n`)。动态分组恰好填满 `top_n`
Expand Down
9 changes: 3 additions & 6 deletions docs/zh/sharded-snapshotdb.md
Original file line number Diff line number Diff line change
Expand Up @@ -200,22 +200,19 @@ selector。输出 `scope` 标识实际 DB/device/slice、受限真实区间、
(单库返回 `None`);其 `paths(QueryScope(...))` 可寻址跨片范围的全部文件,但不执行查询。
默认**每次独立 inspect 都完整校验**最终化合同,并对 manifest/成员取内容哈希。文件元数据
本身不是内容完整性证明:未知或低精度文件系统可能在写入后仍返回相同 `ctime_ns`。
每次查询(包括多步报告中的每个查询步骤)都会再次调用 `inspect`。报告目前不会跨步骤
固定同一个 `ResolvedDataset`,也不会调用 `ResolvedDataset.require_unchanged()`;
没有额外的报告级内容哈希守卫。
已固定 generation 的报告操作默认也重新哈希内容,但不重复逐行校验。

`PT_SNAP_DATASET_CACHE=immutable` 显式启用最多 16 个已校验 generation 的进程级 LRU,
不保留连接。仅在你控制最终化、不可变发布,且信任文件系统内核 change-time 语义时启用。
**可能被原地修改的文件、live writer、未知/低精度/网络文件系统或可能回滚的文件系统快照
不得启用**。这是调用方责任合同,不代表工具已检测文件系统能力。不设置该变量即保留强校验,
未知取值也使用强校验;Windows 或没有可用 POSIX change token 的平台始终使用强校验。

显式启用后,新建 resolver/service 可复用已校验结果。每次 inspect 缓存命中仍验证
显式启用后,新建 resolver/service 可复用已校验结果。每次命中及报告 generation 检查仍验证
规范路径、sidecar、原生成员集合,以及文件的设备号、inode、size、mtime、**ctime**、mode
和硬链接数。这些保守检查可检测常规替换及恢复 mtime 的原地修改,但不能代替不可变文件
合同。固定分片数时,显式启用后的命中不扫描行、不做整文件哈希,工作量为 O(分片数),
不随 event/block 数增长。报告中的查询步骤也适用这种仅检查元数据的复用;命中时不会
重新哈希内容。默认 inspect 及未命中仍与内容规模成正比。显式 `validate_dataset`
不随 event/block 数增长。默认 inspect 及未命中仍与内容规模成正比。显式 `validate_dataset`
/ `validate_native_dataset` 始终完整校验。原生准入保留校验前后两遍哈希,fingerprint 直接
复用已验证摘要,不再做第三遍成员哈希。

Expand Down
24 changes: 15 additions & 9 deletions skills/pt-snap-memory-peak-breakdown/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -244,19 +244,25 @@ pt-snap report peak-memory "<DB>" --device <DEVICE> --metric reserved --include-

Keep the JSON in the response. The report composes these product templates:

- `memory_peak` finds the active, allocated, and reserved peak values and event
IDs.
- `allocator_gap` returns each metric's counters and gaps at that same metric's
peak event.
- `allocator_gap` runs once to find the active, allocated, and reserved peak
values and event IDs, plus each metric's same-event counters and gaps. The
report projects its six-field `peak` section from this result, preserving
`memory_peak` semantics without running a separate `memory_peak` query.
- `active_memory_callstack_at_event` groups blocks active at the report's
selected metric event by allocation callstack.

From the three results, record each metric's own peak value and event ID. Peak
ties resolve to the earliest event ID. Record same-event counters and gaps from
Both steps share one resolved database/device, budget, and dataset generation.
If the selected metric has no peak value or event ID, the report returns empty
attribution without running the attribution query; never invent an event for an
empty range or an all-NULL counter. Other metrics can still have valid peaks.

From the three metric reports, record each metric's own peak value and event ID.
Peak ties resolve to the earliest event ID. Record same-event counters and gaps from
`allocator_gap`; never subtract independently occurring peak values as if they
occurred together. The repeated `peak` and `allocator_gap` sections should agree
across all three reports; the metric-specific `event_id` and `callstack_groups`
are the selected-event breakdown.
occurred together. The repeated `peak` sections should agree across all three
reports. For metrics with a valid peak, the `allocator_gap` sections also agree;
an empty-attribution report has `allocator_gap: null`. The metric-specific
`event_id` and `callstack_groups` are the selected-event breakdown.

Read report `has_more`, `truncated`, `total_is_exact`, and `effective_params`
before claiming a complete composition. A full dynamic `top_n` window means
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -244,19 +244,25 @@ pt-snap report peak-memory "<DB>" --device <DEVICE> --metric reserved --include-

Keep the JSON in the response. The report composes these product templates:

- `memory_peak` finds the active, allocated, and reserved peak values and event
IDs.
- `allocator_gap` returns each metric's counters and gaps at that same metric's
peak event.
- `allocator_gap` runs once to find the active, allocated, and reserved peak
values and event IDs, plus each metric's same-event counters and gaps. The
report projects its six-field `peak` section from this result, preserving
`memory_peak` semantics without running a separate `memory_peak` query.
- `active_memory_callstack_at_event` groups blocks active at the report's
selected metric event by allocation callstack.

From the three results, record each metric's own peak value and event ID. Peak
ties resolve to the earliest event ID. Record same-event counters and gaps from
Both steps share one resolved database/device, budget, and dataset generation.
If the selected metric has no peak value or event ID, the report returns empty
attribution without running the attribution query; never invent an event for an
empty range or an all-NULL counter. Other metrics can still have valid peaks.

From the three metric reports, record each metric's own peak value and event ID.
Peak ties resolve to the earliest event ID. Record same-event counters and gaps from
`allocator_gap`; never subtract independently occurring peak values as if they
occurred together. The repeated `peak` and `allocator_gap` sections should agree
across all three reports; the metric-specific `event_id` and `callstack_groups`
are the selected-event breakdown.
occurred together. The repeated `peak` sections should agree across all three
reports. For metrics with a valid peak, the `allocator_gap` sections also agree;
an empty-attribution report has `allocator_gap: null`. The metric-specific
`event_id` and `callstack_groups` are the selected-event breakdown.

Read report `has_more`, `truncated`, `total_is_exact`, and `effective_params`
before claiming a complete composition. A full dynamic `top_n` window means
Expand Down
Loading
Loading