Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,6 +143,10 @@ pytest # Run all tests
black --check . && ruff check . # Check formatting and lint
```

For the regular suite use `pytest -m 'not slow'`. Default-capacity scale and
non-editable installed exporter tests are described in the
[dataset acceptance guide](docs/en/dataset-acceptance.md).

### Building distributions

Build from a fresh checkout or a new worktree at the intended commit, with no
Expand Down
3 changes: 3 additions & 0 deletions README_zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,9 @@ pytest # 运行所有测试
black --check . && ruff check . # 检查格式和 lint
```

常规套件可用 `pytest -m 'not slow'`。默认容量规模及非 editable 安装包导出测试见
[数据集验收指南](docs/zh/dataset-acceptance.md)。

### 构建分发包

请从目标提交的全新 checkout 或新 worktree 构建,确保没有已有的 `build/` 和
Expand Down
2 changes: 2 additions & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
| Importing snapshots | [en/quickstart.md](en/quickstart.md) |
| Focus Management | [en/focus-management.md](en/focus-management.md) |
| Querying | [en/querying.md](en/querying.md) |
| Dataset correctness and default-scale acceptance | [en/dataset-acceptance.md](en/dataset-acceptance.md) |
| Reports | [en/querying.md](en/querying.md) |
| Splitting Snapshots | [en/splitting.md](en/splitting.md) |
| Agent skills | [en/skills.md](en/skills.md) |
Expand All @@ -26,6 +27,7 @@
| 导入快照 | [zh/quickstart.md](zh/quickstart.md) |
| Focus 管理 | [zh/focus-management.md](zh/focus-management.md) |
| 运行查询 | [zh/querying.md](zh/querying.md) |
| 数据集正确性与默认规模验收 | [zh/dataset-acceptance.md](zh/dataset-acceptance.md) |
| 分析报告 | [zh/querying.md](zh/querying.md) |
| 拆分快照 | [zh/splitting.md](zh/splitting.md) |
| Agent skill | [zh/skills.md](zh/skills.md) |
Expand Down
95 changes: 95 additions & 0 deletions docs/en/dataset-acceptance.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
# Dataset correctness and default-scale acceptance

Dataset tests distinguish three sources of evidence. Passing a synthetic SQL
fixture is not evidence that a producer, collector, or exporter ran.

## Regular correctness gate

`tests/core/test_dataset_realistic.py` runs eight fixed seeds and twenty query
shapes per seed against a standalone reference. Each shard contains only blocks
allocated within its window and blocks still alive on entry. Completed lifetimes
do not appear in later windows; global allocation/free-completion IDs remain
unchanged. Cases include address reuse, a pending free crossing a window,
preexisting/static blocks, NULL/empty/whitespace stacks, a literal missing-stack
label, sorting, pagination, filters and dynamic `top_n` truncation.

Comparisons are exact, including four-decimal percentages. The minimal 1-byte /
127-byte case requires `0.7813`; an 8-MiB scale variant also checks six-decimal
GiB rounding. The contract is exact parity with the active SQLite runtime's
arithmetic and `ROUND`, not a mathematical half-up approximation. Near-half
3/2,000,000 and 7/2,000,000 proportions, nearby values and large GiB conversions
are compared with standalone SQL on that same runtime, since SQLite versions
can differ in how they format binary doubles. Scalar rounding uses short-lived,
explicitly closed in-memory connections without source reads or merge tables;
timeouts during either rounding pass also verify connection/cursor cleanup. Dataset-only provenance is excluded
from common-schema comparisons. Point-event stored `state` is an observation,
not state at that event; global block comparisons use the latest containing
observation for both sides.

A separate two-device native-v2 case executes actual replay/import, then attaches
an explicitly constructed ordered-frame extension. Equal local stack IDs in
different members must not merge different ordered arrays. Order, repeats and
extra raw-frame fields are checked against input arrays. Common event/block
fields are compared with standalone SQL; the optional extension does not claim
that the native writer emits ordered frames.

```bash
pytest tests/test_fixture_provenance.py
pytest tests/core/test_dataset_realistic.py tests/core/test_dataset_attribution.py tests/core/test_dataset_global.py
```

## Slow scale and installed export gates

`tests/core/test_dataset_scale_acceptance.py` is marked `slow` and contains:

- Synthetic compatibility-v1 SQLite: 1,000,000 real events in two default-sized
500,000-event members, with window-correct lifecycle membership. Global peak,
gap, event/allocation pages, block pages, terminal leak candidates and stack
aggregation must succeed
using the real default work budgets. Unbounded materialization must still fail
at the genuine row ceiling. Budgets are not monkeypatched down or disabled.
Ordered-frame aggregation may still need the bounded raw-frame fallback; this
gate does not claim all large ordered-frame or unlimited-output queries succeed.
- A separate 10,000-event / 8-KiB-inline-stack dataset checks aggregate/small-page
success and genuine 64-MiB full-output rejection without creating an 8-GB file.
- A trusted, test-owned snapshot with 500,002 alloc/free-request/free-completion
events goes through a non-editable installed `pt-snap import --format msinsight`
with **no capacity override**. The resulting 500,000 + 2 member layout and
cross-boundary lifecycle membership are checked before installed CLI/API peak,
page, terminal-candidate, point-event and report queries. Artifact hashes must
remain unchanged during queries.

Build the wheel from the intended clean checkout using the
[distribution instructions](../../README.md#building-distributions), then run:

```bash
pytest tests/test_fixture_provenance.py
PT_SNAP_ACCEPTANCE_WHEEL="$(realpath dist/<built-wheel>.whl)" \
pytest tests/core/test_dataset_scale_acceptance.py --junitxml=dataset-acceptance.xml
```

Replace `<built-wheel>` with the actual wheel name. The test installs that exact
wheel into its temporary directory using `--no-deps --no-index`; subprocesses
clear source `PYTHONPATH` and verify that `pt_snap_cli` resolves inside the
installation. Runtime dependencies are reused from the test interpreter: this
is not a fresh dependency-resolution test. Without `PT_SNAP_ACCEPTANCE_WHEEL`,
only the installed test is explicitly skipped; that skip is not E2E acceptance.
Use `pytest -m 'not slow'` for the regular suite.

The installed test writes `acceptance.json` under its pytest temporary directory
and a `dataset_acceptance` JUnit property containing the wheel/source SHA-256,
package path, event counts, import duration and query assertions. Duration is
observational; there is no fixed timing assertion or RSS promise.

## Evidence limits

The [historical performance baseline](../dataset-performance-baseline.json)
uses a 581,140-byte reviewed snapshot and capacity 2,000. Its sharded run has
8,092 real events across five members. It remains useful historical evidence,
but cannot demonstrate default-capacity scalability.

The new exporter gate exercises the real first-party replay/export/query chain
on inspectable test-owned input. It does not execute live PyTorch/NPU collection,
the upstream msinsight producer, or its GUI. Those acceptance steps remain
separate. No new committed pickle is introduced. The provenance gate runs before
collection; the test-owned temporary input is written during the test.
7 changes: 7 additions & 0 deletions docs/en/querying.md
Original file line number Diff line number Diff line change
Expand Up @@ -161,6 +161,13 @@ only when the caller guarantees immutable finalized files and reliable filesyste
change times; see [dataset validation and cache contracts](sharded-snapshotdb.md).
No temporary merge DB, repair or artifact write.

The compatible exporter defaults to **500000 real events per device shard**.
That storage capacity is separate from the **100000-row / 64-MiB query work
budget**: even one default-sized shard can exceed it when a result is materialized
without a sufficient filter or page. Use supported aggregates, bounded pages or
narrower event ranges; a small requested output does not waive source-work limits.
See [default-scale acceptance and evidence limits](dataset-acceptance.md).

Global peak queries fetch scalar range evidence and at most three counter rows per
shard. Event/allocation pages apply SQL filters and ordering per shard, merge at most
`offset + limit` candidates per shard, and count matches separately. Range evidence
Expand Down
73 changes: 73 additions & 0 deletions docs/zh/dataset-acceptance.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
# 数据集正确性及默认规模验收

数据集测试区分三类证据。合成 SQL 夹具通过,不代表生产端、采集器或导出器已运行。

## 常规正确性门禁

`tests/core/test_dataset_realistic.py` 对八个固定种子各运行二十种查询形态,并与单库
参考结果比较。每片仅包含窗口内新分配及进入窗口时仍存活的 block;已完成释放的
生命周期不进入后续窗口,全局分配/完成释放 ID 保持不变。覆盖地址复用、跨窗口
pending free、preexisting/static、NULL/空/空白栈、字面量缺失栈标签、排序、分页、
筛选及动态组 `top_n` 截断。

所有比较严格相等,包括四位小数百分比。1-byte / 127-byte 最小用例要求 `0.7813`,
8-MiB 倍率另检查六位小数 GiB 舍入。契约是与当前 SQLite 运行时的运算及 `ROUND`
严格一致,不是数学上的 half-up 近似。3/2,000,000、7/2,000,000 比例、临近值及大数
GiB 转换都以同一运行时的单库 SQL 为参考,因为 SQLite 版本间的二进制浮点格式化
可能不同。标量舍入使用短生命周期且显式关闭的内存连接,不读取来源或创建合并表;
两个舍入阶段的超时测试也验证连接/cursor 关闭。公共 schema 比较排除
数据集独有来源元数据。定点 block 的存储 `state` 是观察值,不是该事件时刻状态;
全局 block 两侧都采用最后一个包含该生命周期的窗口观察。

另一个双设备原生 v2 用例执行真实 replay/import,再附加明确构造的有序 frame 扩展。
不同成员中的相同局部栈 ID 不得合并不同有序数组;顺序、重复项及额外原始 frame
字段都与输入数组核对。公共事件/block 字段与单库 SQL 比较;该扩展夹具不声称原生
writer 会生成有序 frames。

```bash
pytest tests/test_fixture_provenance.py
pytest tests/core/test_dataset_realistic.py tests/core/test_dataset_attribution.py tests/core/test_dataset_global.py
```

## slow 规模及安装包导出门禁

`tests/core/test_dataset_scale_acceptance.py` 标记为 `slow`,包括:

- 合成兼容 v1 SQLite:1,000,000 个真实事件,两个默认大小的 500,000-event 成员,
block 成员符合生命周期窗口。全局 peak、gap、event/allocation 分页、block 分页及
终片泄漏候选及栈聚合必须在真实默认工作预算内成功;无界物化仍需触发真实行数上限。测试不调低
或关闭预算。有序 frame 聚合可能仍需有界原始 frame 回退路径;此门禁不声称所有
大规模有序 frame 或无界输出查询均可成功。
- 独立 10,000-event / 8-KiB 内联栈数据集验证聚合/小页成功,以及完整输出触发真实
64-MiB 上限,无需创建 8-GB 数据文件。
- 受信、测试自行构造的 500,002 个 alloc/free-request/free-completion 事件快照,
经非 editable 安装的 `pt-snap import --format msinsight`,**不覆盖容量默认值**。
先核对 500,000 + 2 成员布局及跨界生命周期,再通过该安装包的 CLI/API 验证峰值、
分页、终片候选、定点事件及报告。查询前后产物 hash 必须不变。

按照[分发包构建说明](../../README_zh.md#构建分发包),从目标干净 checkout 构建 wheel:

```bash
pytest tests/test_fixture_provenance.py
PT_SNAP_ACCEPTANCE_WHEEL="$(realpath dist/<built-wheel>.whl)" \
pytest tests/core/test_dataset_scale_acceptance.py --junitxml=dataset-acceptance.xml
```

将 `<built-wheel>` 替换为实际名称。测试以 `--no-deps --no-index` 将该确切 wheel 安装到
临时目录,子进程清除源码 `PYTHONPATH` 并核对 `pt_snap_cli` 来自该安装目录。运行依赖
复用测试解释器环境,不代表全新依赖解析测试。未设置 `PT_SNAP_ACCEPTANCE_WHEEL` 时,
仅安装包用例明确 skip;skip 不算 E2E 通过。常规套件可用 `pytest -m 'not slow'`。

安装包用例在 pytest 临时目录写 `acceptance.json`,并记录 `dataset_acceptance` JUnit
属性,包括 wheel/source SHA-256、包路径、事件数量、导入耗时及查询断言。耗时只是
单次观察,不设固定时限断言,也不承诺进程 RSS。

## 证据边界

[历史性能基线](../dataset-performance-baseline.json) 使用 581,140-byte 已审核快照及
容量 2,000;分片模式仅 8,092 个真实事件、五片。保留此历史证据,但它不能证明默认
容量可扩展性。

新增导出门禁在可审查的测试输入上执行真实第一方 replay/export/query 链路;不执行
实时 PyTorch/NPU 采集、上游 msinsight 生产端或 GUI,这些验收仍独立。未增加任何提交
到仓库的 pickle。provenance 门禁在收集测试前运行;临时受信输入在测试期间自行写入。
7 changes: 6 additions & 1 deletion docs/zh/querying.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,13 +130,18 @@ exactness 标记诚实。`has_more/truncated` 描述窗口,不表示来源/范
数据集读取先分离批次再查外片来源,使用借用的有界 Context LRU。所有片、来源、分组、
精确总数共用递减 deadline;报告的峰值选择与归因明确共用**一个**预算。累计取回/输出工作上限为
**100000 行 / 64 MiB 序列化值**,含重复来源读取;超限整体失败,不返回冒充完整的部分
结论。Python 物化/排序由工作上限约束,**不由 max_rows 约束**;不是进程 RSS 上限,Python/
结论。物化/排序由工作上限约束,**不由 max_rows 约束**;不是进程 RSS 上限,Python/
SQLite、一个批次及单个 cell 有额外开销。默认仍对 manifest/成员执行完整校验和 hash,
并共用该 deadline:hash 分块、Python 行批次及 SQLite progress handler 都检查取消。
单个文件系统调用不提供 OS 级硬抢占。仅当调用方保证最终产物不可变且文件系统变更时间
可靠时,才可通过 `PT_SNAP_DATASET_CACHE=immutable` 显式启用进程内校验复用;参见
[数据集校验与缓存合同](sharded-snapshotdb.md)。不生成合并临时 DB、不修复或写产物。

兼容导出器默认**每设备每片 500000 个真实事件**。该存储容量与 **100000 行 / 64 MiB
查询工作预算**独立:即使只有一个默认大小的分片,无足够筛选或分页的结果物化也可能
超限。请使用受支持的聚合、有界分页或较窄事件范围;请求输出较少不代表可忽略来源
读取预算。详见[默认规模验收及证据边界](dataset-acceptance.md)。

全局峰值查询每片只取范围汇总及最多三条计数器记录。event/allocation 在各片 SQL 中筛选、
排序,每片最多取 `offset + limit` 个候选后全局归并,并单独统计匹配总数;实际范围证据在
非范围筛选前计算。普通文本栈统计先合并各片完整分组,再应用全局阈值,平均值由总大小
Expand Down
47 changes: 38 additions & 9 deletions src/pt_snap_cli/core/dataset_attribution.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,13 @@

from __future__ import annotations

import sqlite3
from collections.abc import Iterator
from contextlib import closing, contextmanager
from typing import cast

from pt_snap_cli.core.dataset_sources import DatasetSourceResolver, EventSource
from pt_snap_cli.core.errors import QueryExecutionError
from pt_snap_cli.core.models import QueryResult
from pt_snap_cli.core.stack_summary import summarize_stacks

Expand Down Expand Up @@ -139,12 +143,16 @@ def event_attribution(
)
selected.sort(key=_rank)
denominator = sum(cast(int, row["size_bytes"]) for row in selected)
for row in selected:
row["percent_of_active_blocks"] = (
round(cast(int, row["size_bytes"]) * 100.0 / denominator, 4)
if denominator
else None
)
# Use the same SQLite arithmetic/ROUND implementation as standalone SQL.
# A Decimal conversion of a binary float changes some near-half values.
# This connection evaluates scalars only: no source reads or merge tables.
with _rounding_cursor(sources) as cursor:
for row in selected:
sources.budget.remaining()
row["percent_of_active_blocks"] = cursor.execute(
"SELECT ROUND(? * 100.0 / NULLIF(?, 0), 4)",
(row["size_bytes"], denominator),
).fetchone()[0]
offset = 0
has_more = len(selected) < len(all_rows) - offset or rank_full
if max_rows is not None and max_rows > 0:
Expand Down Expand Up @@ -243,9 +251,16 @@ def _groups(
cast(int, group["stack_event_id"]), cast(int, source.event["id"])
)
rows = list(grouped.values())
for row in rows:
row["size_gib"] = round(cast(int, row["size_bytes"]) / 1073741824.0, 6)
row["requested_gib"] = round(cast(int, row["requested_bytes"]) / 1073741824.0, 6)
# Reuse one scalar-only engine for this pass and close it on timeout as well
# as success. Derived values do not add source IO; final output is still
# charged by event_attribution's existing consume(selected) boundary.
with _rounding_cursor(sources) as cursor:
for row in rows:
sources.budget.remaining()
row["size_gib"], row["requested_gib"] = cursor.execute(
"SELECT ROUND(? / 1073741824.0, 6), ROUND(? / 1073741824.0, 6)",
(row["size_bytes"], row["requested_bytes"]),
).fetchone()
rows.sort(key=_rank)
return rows

Expand All @@ -261,3 +276,17 @@ def summarize_dataset_stacks(rows: list[dict[str, object]], budget: int) -> None
for row, (identity, kind) in zip(rows, identities, strict=True):
row["stack_id"] = identity
row["stack_kind"] = kind


@contextmanager
def _rounding_cursor(sources: DatasetSourceResolver) -> Iterator[sqlite3.Cursor]:
"""Keep scalar arithmetic failures on the same domain boundary as shard SQL."""
try:
with (
closing(sqlite3.connect(":memory:")) as connection,
closing(connection.cursor()) as cursor,
):
yield cursor
except (sqlite3.Error, OverflowError) as exc:
sources.budget.remaining() # Preserve a deadline that expired during SQLite work.
raise QueryExecutionError(str(exc)) from exc
Loading
Loading