Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion .github/scripts/run_dataset_acceptance.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,11 @@
TEST_NODE = f"tests/core/test_dataset_scale_acceptance.py::{TEST_NAME}"
# Extend this tuple alongside new mandatory nightly gates. Never list future
# files that have not landed: the runner rejects any missing configured node.
REQUIRED_TEST_NODES = (TEST_NODE,)
BOUNDARY_TEST_NODE = (
"tests/core/test_dataset_scale_acceptance.py::"
"test_installed_default_export_short_final_shard_boundary"
)
REQUIRED_TEST_NODES = (TEST_NODE, BOUNDARY_TEST_NODE)


class AcceptanceError(RuntimeError):
Expand Down
28 changes: 23 additions & 5 deletions .github/workflows/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,11 +45,29 @@ python .github/scripts/run_dataset_acceptance.py \
--wheel dist/<built-wheel>.whl --output-dir .tmp/dataset-acceptance
```

This tier initially exercises the existing 500,002-event `pip --target` acceptance
test. It does not claim a clean venv, two full 500,000-event shards, complete CLI
template coverage, live agents or upstream GUI interoperability. The ordinary
PR tier excludes `nightly` tests; the required nightly/manual runner must never
skip them. Manually dispatch the exact published branch/commit and
The required installed-wheel case creates a clean venv without system packages
and runs its actual `pt-snap` console script from isolated HOME/CWD. Its trusted,
test-owned input has exactly 1,000,000 events; the default exporter produces two
full 500,000-event shards. All installed catalog entries are classified: currently
11 dataset-supported templates succeed with installed CLI/API equivalence and
independent known-result checks, while `event_frames` and
`active_memory_frame_tree_at_event` return explicit unsupported-dataset errors.
Both report commands run: `peak-memory` succeeds and `memory-tree` preserves the
corresponding dataset rejection. The allocation/free-request/free-completion
boundary assertions remain covered, alongside static occupancy, offset pages,
terminal survivors and unchanged source/manifest/member hashes. A second required
nightly case separately imports 500,002 events and proves the original final
short shard has exactly two real events, with the crossing completed lifetime
and the final survivor. Its evidence is stored in its own subdirectory so the
two gates cannot overwrite each other's command logs. This adds one half-million
event import plus five query subprocesses to the nightly job. Existing
synthetic scale/default-budget regressions remain separate source-level tests.
`PT_SNAP_ACCEPTANCE_WHEELHOUSE` optionally supplies dependencies for an offline
clean-venv install. Timings are recorded without machine-speed thresholds; this
does not claim live agents, live hardware capture or upstream GUI interoperability.

The ordinary PR tier excludes `nightly` tests; the required nightly/manual runner
must never skip them. Manually dispatch the exact published branch/commit and
verify the run's head SHA and summary; a PR run alone is not nightly evidence.

## Release Workflow
Expand Down
46 changes: 28 additions & 18 deletions docs/en/dataset-acceptance.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,12 +128,17 @@ separate empty/static/dynamic/mixed regressions across both layouts and reducers
gate does not claim all large ordered-frame or unlimited-output queries succeed.
- A separate 10,000-event / 8-KiB-inline-stack dataset checks aggregate/small-page
success and genuine 64-MiB full-output rejection without creating an 8-GB file.
- A trusted, test-owned snapshot with 500,002 alloc/free-request/free-completion
events goes through a non-editable installed `pt-snap import --format msinsight`
with **no capacity override**. The resulting 500,000 + 2 member layout and
cross-boundary lifecycle membership are checked before installed CLI/API peak,
page, terminal-candidate, point-event and report queries. Artifact hashes must
remain unchanged during queries.
- A trusted, test-owned snapshot with 1,000,000 alloc/free-request/free-completion
events goes through the clean-venv installed `pt-snap import --format msinsight`
with **no capacity override**, producing two full 500,000-event members. All
installed catalog templates are classified: 11 dataset-supported templates
have CLI/API parity and independent result checks; `event_frames` and
`active_memory_frame_tree_at_event` retain explicit dataset rejection. The
`peak-memory` report succeeds; `memory-tree` retains its dataset rejection.
- A second required installed-wheel case preserves the original 500,002-event
workload and its 500,000 + 2 member layout. It verifies the crossing completed
lifetime, final survivor, pages and point-event results. Both installed cases
require unchanged input, manifest and member hashes after queries.

Build the wheel from the intended clean checkout using the
[distribution instructions](../../README.md#building-distributions), then run:
Expand All @@ -144,18 +149,23 @@ PT_SNAP_ACCEPTANCE_WHEEL="$(realpath dist/<built-wheel>.whl)" \
pytest tests/core/test_dataset_scale_acceptance.py --junitxml=dataset-acceptance.xml
```

Replace `<built-wheel>` with the actual wheel name. The test installs that exact
wheel into its temporary directory using `--no-deps --no-index`; subprocesses
clear source `PYTHONPATH` and verify that `pt_snap_cli` resolves inside the
installation. Runtime dependencies are reused from the test interpreter: this
is not a fresh dependency-resolution test. Without `PT_SNAP_ACCEPTANCE_WHEEL`,
only the installed test is explicitly skipped; that skip is not E2E acceptance.
Use `pytest -m 'not slow'` for the regular suite.

The installed test writes `acceptance.json` under its pytest temporary directory
and a `dataset_acceptance` JUnit property containing the wheel/source SHA-256,
package path, event counts, import duration and query assertions. Duration is
observational; there is no fixed timing assertion or RSS promise.
Replace `<built-wheel>` with the actual wheel name. Each installed case creates
its own fresh venv without system packages, installs the exact wheel and runtime
dependencies, clears source `PYTHONPATH`, and checks that the package and
runtime dependencies resolve inside that venv. Dependency installation may use
the package index. Set `PT_SNAP_ACCEPTANCE_WHEELHOUSE` to a directory containing
compatible dependency wheels for an offline `--no-index --find-links` install.
Without `PT_SNAP_ACCEPTANCE_WHEEL`, both installed cases are explicitly skipped;
those skips are not E2E acceptance. The full PR tier uses `pytest -m 'not nightly'`;
the compatibility tier uses `pytest -m 'not slow and not nightly'`.

The million-event installed case writes `acceptance.json` and a
`dataset_acceptance` JUnit property with wheel/source hashes, runtime identity,
event counts and command/query evidence. The short-final-shard case stores its
own evidence separately. `PT_SNAP_ACCEPTANCE_EVIDENCE_DIR` selects a persistent
evidence directory; otherwise pytest temporary directories are used. The
required nightly runner rejects a missing or nonpassing case. Durations are
observational; these installed-wheel cases have no fixed timing or RSS promise.

## Evidence limits

Expand Down
34 changes: 22 additions & 12 deletions docs/zh/dataset-acceptance.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,10 +103,15 @@ fallback 三条路径的这些契约。`top_n=0` 完整性另有空/static/dynam
大规模有序 frame 或无界输出查询均可成功。
- 独立 10,000-event / 8-KiB 内联栈数据集验证聚合/小页成功,以及完整输出触发真实
64-MiB 上限,无需创建 8-GB 数据文件。
- 受信、测试自行构造的 500,002 个 alloc/free-request/free-completion 事件快照,
经非 editable 安装的 `pt-snap import --format msinsight`,**不覆盖容量默认值**。
先核对 500,000 + 2 成员布局及跨界生命周期,再通过该安装包的 CLI/API 验证峰值、
分页、终片候选、定点事件及报告。查询前后产物 hash 必须不变。
- 受信、测试自行构造的 1,000,000 个 alloc/free-request/free-completion 事件快照,
经干净 venv 中安装的 `pt-snap import --format msinsight`,**不覆盖容量默认值**,
生成两个完整的 500,000-event 成员。所有已安装目录模板均被分类:11 个支持数据集的
模板核对 CLI/API 一致性及独立结果;`event_frames` 和
`active_memory_frame_tree_at_event` 保留明确的数据集拒绝。`peak-memory` 报告成功,
`memory-tree` 保留数据集拒绝。
- 第二个必需的安装包用例保留原有 500,002-event 工作负载及 500,000 + 2 成员布局,
验证跨界完成生命周期、最终存活分配、分页及定点事件结果。两个安装包用例均要求
查询前后输入、manifest 及成员 hash 不变。

按照[分发包构建说明](../../README_zh.md#构建分发包),从目标干净 checkout 构建 wheel:

Expand All @@ -116,14 +121,19 @@ PT_SNAP_ACCEPTANCE_WHEEL="$(realpath dist/<built-wheel>.whl)" \
pytest tests/core/test_dataset_scale_acceptance.py --junitxml=dataset-acceptance.xml
```

将 `<built-wheel>` 替换为实际名称。测试以 `--no-deps --no-index` 将该确切 wheel 安装到
临时目录,子进程清除源码 `PYTHONPATH` 并核对 `pt_snap_cli` 来自该安装目录。运行依赖
复用测试解释器环境,不代表全新依赖解析测试。未设置 `PT_SNAP_ACCEPTANCE_WHEEL` 时,
仅安装包用例明确 skip;skip 不算 E2E 通过。常规套件可用 `pytest -m 'not slow'`。

安装包用例在 pytest 临时目录写 `acceptance.json`,并记录 `dataset_acceptance` JUnit
属性,包括 wheel/source SHA-256、包路径、事件数量、导入耗时及查询断言。耗时只是
单次观察,不设固定时限断言,也不承诺进程 RSS。
将 `<built-wheel>` 替换为实际名称。每个安装包用例创建独立且不含系统包的全新 venv,
安装该确切 wheel 及运行依赖,清除源码 `PYTHONPATH`,并核对产品包和运行依赖均来自
该 venv。依赖安装可能访问包索引。可将 `PT_SNAP_ACCEPTANCE_WHEELHOUSE` 指向包含兼容
依赖 wheel 的目录,以 `--no-index --find-links` 离线安装。
未设置 `PT_SNAP_ACCEPTANCE_WHEEL` 时,两个安装包用例均明确 skip;skip 不算 E2E 通过。
完整 PR 层使用 `pytest -m 'not nightly'`;兼容层使用
`pytest -m 'not slow and not nightly'`。

百万事件安装包用例写入 `acceptance.json` 和 `dataset_acceptance` JUnit 属性,包含
wheel/source hash、运行环境身份、事件数量及命令/查询证据。短终片用例单独保存其证据。
`PT_SNAP_ACCEPTANCE_EVIDENCE_DIR` 可指定持久证据目录,否则使用 pytest 临时目录。
必需的 nightly runner 拒绝缺失或未通过的用例。耗时仅为观察记录;这些安装包用例
不设固定时限,也不承诺进程 RSS。

## 证据边界

Expand Down
35 changes: 35 additions & 0 deletions tests/core/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,3 +37,38 @@ Implementation: [core services](../../src/pt_snap_cli/core/AGENTS.md).
Run `pytest tests/core` or the owning file from the repository root. Follow the
parent's fixture gate before import/split cases and resource ownership rules for
all cached services. Cross-runtime changes additionally use `tests/snapshot/`.

## Installed Dataset Scale Acceptance

`test_dataset_scale_workload.py` checks the test-owned allocator workload and
arithmetic oracle on 100 events. `test_dataset_scale_acceptance.py` retains the
synthetic default-budget gates and has one opt-in wheel acceptance node:
`test_installed_trusted_producer_default_export_cli_and_api`.

Set `PT_SNAP_ACCEPTANCE_WHEEL` to an absolute wheel path. The node creates a fresh
venv without system packages, installs the wheel and its dependencies, clears
source/focus/query-budget overrides, and executes its actual `pt-snap` console
script in isolated HOME/CWD. Optional `PT_SNAP_ACCEPTANCE_WHEELHOUSE` enables
fully offline dependency installation with `--no-index --find-links`.
`PT_SNAP_ACCEPTANCE_EVIDENCE_DIR` preserves subprocess commands, exit codes,
timings, stdout/stderr, versions, wheel/source/member hashes, and acceptance JSON.
The same acceptance is stored in the JUnit `dataset_acceptance` property (use
`-o junit_family=xunit1`). Timings are observations, not machine-speed thresholds.

The real importer exports exactly 1,000,000 events using the unchanged 500,000
default capacity. The complete installed catalog must be classified: currently
11 dataset-supported templates succeed with CLI/API result parity and independent
known-result assertions; `event_frames` and `active_memory_frame_tree_at_event`
are explicit dataset rejections. Both report commands run through the console:
`peak-memory` succeeds, while `memory-tree` preserves its documented unsupported
dataset contract. Source and every exported member remain unchanged. This is
trusted test-owned producer/replay acceptance, not live capture or upstream GUI
acceptance. Do not turn a missing wheel/skip into a successful nightly gate.

The separate required nightly
`test_installed_default_export_short_final_shard_boundary` retains the original
500,002-event shape: 500,000 real events followed by exactly two. It checks exact
terminal member lifetimes and installed CLI/API completion/survival boundaries,
including unchanged hashes. Its `dataset_boundary_acceptance` JUnit property and
per-test evidence subdirectory preserve independent results without overwriting
the million-event gate's evidence. Both cases are mandatory in the runner.
Loading
Loading