Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 42 additions & 0 deletions docs/en/dataset-acceptance.md
Original file line number Diff line number Diff line change
Expand Up @@ -169,3 +169,45 @@ on inspectable test-owned input. It does not execute live PyTorch/NPU collection
the upstream msinsight producer, or its GUI. Those acceptance steps remain
separate. No new committed pickle is introduced. The provenance gate runs before
collection; the test-owned temporary input is written during the test.

## Concurrent readers and process interruption

`tests/core/test_dataset_concurrency.py` runs two and four independent readers
in both threads and spawned processes, against the same synthetic dataset and
through both ATTACH and scratch routes. Barriers follow an actual SQLite read
or scratch insert; all readers hold their own query resources concurrently.
Their complete results must agree, scratch directories must be distinct and
removed on normal completion, and source hashes must stay unchanged. This does
not promise that one analyzer or its SQLite connections can be shared by threads.

`tests/test_dataset_signal_boundaries.py` sends actual POSIX signals to the CLI
after a completed scratch insert. Normal completion and SIGINT unwind product
resources and remove scratch. **Current SIGTERM and SIGKILL terminate the process
without that cleanup: the private `pt-snap-lifecycle-*` directory and `derived.db`
remain.** The tests record this limitation, rather than treating test teardown as
product cleanup or claiming issue #252's no-residue kill goal is satisfied. Source
hashes remain unchanged and no partial query output is published. No signal
handler, startup reclamation, or automatic deletion policy is introduced.

These small deterministic tests run in the regular suite (signal cases are POSIX
only). Timeout/error cleanup is separate from abrupt termination: Python-level
unwinding is not evidence that SIGTERM or SIGKILL runs a `finally` block.

`tests/test_dataset_mutation_acceptance.py` pauses a spawned CLI/API query after
its first shard has actually supplied rows or a count, then changes test-owned
members in place. Both the default strong and explicit immutable cache modes must
reject the changed generation without publishing partial rows/counts. Unchanged
runs succeed with independently expected results. Hash-work assertions quantify
compatibility-v1 cold/warm queries: default queries scan each manifest/member
three/two times respectively (admission plus the new final check). Explicit POSIX
immutable queries scan members once/zero times and the manifest twice/zero times.
The final check itself adds one full scan in default mode, zero in metadata-only
immutable mode, and shares the query's deadline. Non-POSIX immutable requests do
not reuse admission based on creation times: both cold and warm queries scan each
member twice and the manifest three times, including a full final hash guard.
Default strong warm reuse still works on non-POSIX systems. The tests exercise
this conservative policy through a resolver-local simulated non-POSIX case too;
that simulation is not evidence of actual Windows execution. These are work-count assertions,
not fixed wall-time or RSS promises. The guard detects these controlled mutations;
it does not create an atomic filesystem snapshot or prove all change-and-restore
races impossible.
8 changes: 8 additions & 0 deletions docs/en/querying.md
Original file line number Diff line number Diff line change
Expand Up @@ -193,6 +193,14 @@ check cancellation. Individual filesystem calls are not OS-level hard-preempted.
`PT_SNAP_DATASET_CACHE=immutable` explicitly opts into metadata-only validation reuse
only when the caller guarantees immutable finalized files and reliable filesystem
change times; see [dataset validation and cache contracts](sharded-snapshotdb.md).
Every dataset query also verifies that admitted generation before returning rows or
counts, including count-only queries. A detected concurrent change fails the whole
query with a database-integrity error; no partial result is published. By default,
this final guard adds one complete manifest/member hash pass under the same
operation deadline. The explicit immutable mode retains its metadata-only guard
when its existing filesystem requirements hold. This is a change-detection boundary,
not an atomic filesystem snapshot: it does not prove all change-and-restore races
impossible or support querying mutable live SQLite artifacts.
Source artifacts are never repaired or written. A large lifecycle query may create a
private temporary derived database as described below.

Expand Down
33 changes: 33 additions & 0 deletions docs/zh/dataset-acceptance.md
Original file line number Diff line number Diff line change
Expand Up @@ -134,3 +134,36 @@ PT_SNAP_ACCEPTANCE_WHEEL="$(realpath dist/<built-wheel>.whl)" \
新增导出门禁在可审查的测试输入上执行真实第一方 replay/export/query 链路;不执行
实时 PyTorch/NPU 采集、上游 msinsight 生产端或 GUI,这些验收仍独立。未增加任何提交
到仓库的 pickle。provenance 门禁在收集测试前运行;临时受信输入在测试期间自行写入。

## 并发读取与进程中断

`tests/core/test_dataset_concurrency.py` 对同一份合成数据集分别启动两个和四个
独立读取者,覆盖线程、spawn 子进程,以及 ATTACH 和 scratch 两条路径。
屏障位于实际 SQLite 读取或 scratch 插入完成之后,保证所有读取者同时持有各自的
查询资源。完整结果必须一致,scratch 目录必须互不相同且在正常完成后移除,源文件
哈希必须保持不变。这不代表同一个 analyzer 或其 SQLite 连接可以跨线程共享。

`tests/test_dataset_signal_boundaries.py` 在实际 scratch 插入完成后向 CLI 发送
真实 POSIX 信号。正常完成和 SIGINT 会展开产品资源并移除 scratch。
**目前 SIGTERM 和 SIGKILL 会直接终止进程,未执行上述清理,私有
`pt-snap-lifecycle-*` 目录及 `derived.db` 会残留。** 测试明确记录这一限制,
不会把测试清理当作产品清理,也不宣称已经满足 issue #252 的 kill 后无残留目标。
源文件哈希保持不变,不发布部分查询结果。本次未引入信号处理器、启动回收或自动删除策略。

这些小规模、确定性测试进入常规套件(信号用例仅限 POSIX)。超时及错误的清理与
进程被直接终止是不同边界:Python 异常展开不能证明 SIGTERM 或 SIGKILL 会执行
`finally`。

`tests/test_dataset_mutation_acceptance.py` 在 spawn 子进程中的 CLI/API 查询实际
读取第一片的行或计数之后暂停,由父进程原地修改测试自有成员。默认强校验和显式
immutable 缓存模式都必须拒绝已变更的 generation,不发布部分行或计数。
未变更的运行必须成功且符合独立计算的结果。哈希工作量断言明确兼容 v1 的冷/热查询成本:
默认分别对每个 manifest/成员扫描三次/两次(准入加最终检查);POSIX immutable
模式对成员分别扫描一次/零次,对 manifest 分别扫描两次/零次。最终检查本身在默认模式
增加一次完整扫描,在纯元数据 immutable 模式不增加哈希扫描,并共用查询 deadline。
非 POSIX 上的 immutable 请求不会根据创建时间复用准入:冷/热查询都对每个成员扫描
两次、对 manifest 扫描三次,包含完整的最终哈希检查。默认强校验在非 POSIX 上仍可复用
经过内容验证的热准入。测试还通过仅替换 resolver 的平台对象,确定性覆盖这一保守策略;
此模拟不是实际 Windows 执行证据。
这是工作量断言,不是固定耗时或 RSS 保证。检查会检测这些受控变更,但不会创建原子文件系统
快照,也不证明所有先变更再恢复的竞态都能被检测。
10 changes: 8 additions & 2 deletions docs/zh/querying.md
Original file line number Diff line number Diff line change
Expand Up @@ -163,8 +163,14 @@ SQLite、一个批次及单个 cell 有额外开销。默认仍执行强准入
分片数),新 CLI 进程仍冷启动。准入共用该 deadline:hash 分块、Python 行批次及 SQLite progress handler 都检查取消。
单个文件系统调用不提供 OS 级硬抢占。仅当调用方保证最终产物不可变且文件系统变更时间
可靠时,才可通过 `PT_SNAP_DATASET_CACHE=immutable` 显式启用纯元数据校验复用;参见
[数据集校验与缓存合同](sharded-snapshotdb.md)。不修复或写入源产物;较大的生命周期查询
可能生成下述私有临时派生数据库。
[数据集校验与缓存合同](sharded-snapshotdb.md)。
每次数据集查询在返回行或计数之前,都会再次检查已准入的 generation,纯计数查询也不例外。
一旦检测到并发变更,整个查询以数据库完整性错误失败,不发布部分结果。默认情况下,
这一最终检查在同一操作 deadline 内额外扫描一次 manifest/全部成员的完整内容哈希。
显式 immutable 模式在满足原有文件系统要求时,仍使用纯元数据的最终检查。
这是变更检测边界,不是原子文件系统快照:不证明所有先变更再恢复的竞态都能被检测,
也不代表支持查询可变的在线 SQLite 产物。
不修复或写入源产物;较大的生命周期查询可能生成下述私有临时派生数据库。

兼容导出器默认**每设备每片 500000 个真实事件**。该存储容量与 **100000 行 / 64 MiB
查询工作预算**独立:即使只有一个默认大小的分片,无足够筛选或分页的结果物化也可能
Expand Down
8 changes: 4 additions & 4 deletions src/pt_snap_cli/core/query_service.py
Original file line number Diff line number Diff line change
Expand Up @@ -302,10 +302,10 @@ def execute_query(
if isinstance(stack_bytes, int) and stack_bytes >= 0:
summarize_dataset_stacks(result.rows, stack_bytes)
budget.remaining()
if _resolved is not None:
# Inspection establishes the baseline. Reject changed contents
# after each composed step, before any report can be returned.
dataset.require_unchanged(budget)
# Inspection establishes the baseline for standalone queries and
# composed reports alike. Reject mixed-generation rows or counts
# before either adapter can publish any part of the result.
dataset.require_unchanged(budget)
if result.scope is not None:
result.scope["work_budget"] = budget.describe()
return result
Expand Down
105 changes: 105 additions & 0 deletions tests/core/test_dataset_concurrency.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
"""Independent readers overlap real query stages without sharing analyzers."""

import multiprocessing
import queue
import threading

import pytest

from tests.core.dataset_lifecycle_fixtures import build
from tests.dataset_concurrency_support import (
ISOLATED_ENV,
WAIT_SECONDS,
lifecycle_stage,
process_reader,
query_worker,
read_query,
)
from tests.test_dataset_focus import hashes


@pytest.fixture(autouse=True)
def isolate(tmp_path, monkeypatch):
monkeypatch.chdir(tmp_path)
monkeypatch.setenv("HOME", str(tmp_path))
for name in ISOLATED_ENV:
monkeypatch.delenv(name, raising=False)


@pytest.mark.parametrize("readers", [2, 4])
@pytest.mark.parametrize("route", ["attach", "scratch"])
@pytest.mark.parametrize("kind", ["threads", "processes"])
def test_simultaneous_readers_return_complete_identical_results(tmp_path, kind, route, readers):
_, root, _, _, blocks = build(tmp_path, 7)
before = hashes(root)
scratch_parent = tmp_path / "scratch"
scratch_parent.mkdir()
with lifecycle_stage(route, scratch_parent, lambda evidence: None):
expected = read_query(root, route)
assert expected["returned"] == expected["total"] == len(blocks)
assert expected["total_is_exact"] and not expected["has_more"]
assert {row["id"] for row in expected["rows"]} == {block["id"] for block in blocks}

context = multiprocessing.get_context("spawn") if kind == "processes" else threading
ready = context.Barrier(readers + 1)
release = context.Event()
output = context.Queue() if kind == "processes" else queue.Queue()
workers = []

def reached(evidence):
output.put(("stage", evidence))
ready.wait(WAIT_SECONDS)
if not release.wait(WAIT_SECONDS):
raise TimeoutError("Parent did not release the query-stage barrier")

try:
with lifecycle_stage(route, scratch_parent, reached):
for _ in range(readers):
worker = (
context.Process(
target=process_reader,
args=(root, route, scratch_parent, ready, release, output),
)
if kind == "processes"
else threading.Thread(target=query_worker, args=(root, route, output))
)
workers.append(worker)
worker.start()
# Every reader has completed a real SQL operation and holds its own
# connection open. No scheduling speed or arbitrary sleep is evidence.
ready.wait(WAIT_SECONDS)
stages = [output.get(timeout=WAIT_SECONDS) for _ in range(readers)]
assert all(kind == "stage" for kind, _ in stages), stages
identities = {(item["pid"], item["thread"]) for _, item in stages}
assert len(identities) == readers
if route == "scratch":
paths = {item["database"] for _, item in stages}
assert len(paths) == readers
assert all(item["inserted_rows"] > 0 for _, item in stages)
assert len(list(scratch_parent.glob("*/derived.db"))) == readers
else:
assert not list(scratch_parent.iterdir())
assert hashes(root) == before
release.set()
results = [output.get(timeout=WAIT_SECONDS) for _ in range(readers)]
assert all(kind == "result" for kind, _ in results), results
assert all(result == expected for _, result in results)
finally:
release.set()
ready.abort()
for worker in workers:
worker.join(WAIT_SECONDS)
if kind == "processes" and worker.is_alive():
worker.terminate()
worker.join(WAIT_SECONDS)
assert not worker.is_alive()
if kind == "processes":
assert worker.exitcode == 0
worker.close()
if kind == "processes":
output.close()
output.join_thread()
# Only after all product contexts have exited do we check cleanup. The test
# does not remove scratch directories to manufacture this observation.
assert not list(scratch_parent.iterdir())
assert hashes(root) == before
143 changes: 143 additions & 0 deletions tests/dataset_concurrency_support.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,143 @@
"""Real query-stage barriers shared by thread, spawned-process and signal tests.

Hooks observe a completed SQLite read/insert, then block on explicit IPC. They
never replace query results, install signal handlers, or perform product cleanup.
"""

import os
import sys
import threading
import traceback
from contextlib import ExitStack, contextmanager, redirect_stderr, redirect_stdout
from pathlib import Path
from unittest.mock import patch

from pt_snap_cli.api import QueryLimits, SnapshotAnalyzer
from pt_snap_cli.core import dataset_lifecycle_scratch as scratch
from pt_snap_cli.core import dataset_lifecycle_sql as sql
from pt_snap_cli.core.dataset_lifecycle_connection import _AttachedSources

WAIT_SECONDS = 30
QUERY_PARAMS = {"order_by": "id", "order_dir": "ASC", "limit": -1}
ISOLATED_ENV = (
"PT_SNAP_DB_PATH",
"PT_SNAP_DATASET_CACHE",
"PT_SNAP_QUERY_TIMEOUT",
"PT_SNAP_MAX_WORK_ROWS",
"PT_SNAP_MAX_WORK_BYTES",
"PT_SNAP_MAX_SCRATCH_DB_BYTES",
)


@contextmanager
def lifecycle_stage(route, scratch_parent, reached):
original_read = _AttachedSources.read
original_execute = scratch._ScratchSources.execute
original_directory = scratch.tempfile.TemporaryDirectory

def observe(database):
if getattr(database, "_test_stage_reached", False):
return
database._test_stage_reached = True
evidence = {"pid": os.getpid(), "thread": threading.get_ident(), "route": route}
if route == "scratch":
# These are actual product-created bytes, not a synthetic ready file.
cursor = database.connection.execute("PRAGMA database_list")
try:
evidence["database"] = next(row[2] for row in cursor if row[1] == "main")
finally:
cursor.close()
cursor = database.connection.execute("SELECT COUNT(*) FROM lifecycle_state")
try:
evidence["inserted_rows"] = cursor.fetchone()[0]
finally:
cursor.close()
reached(evidence)

def read(database, statement, values=None):
result = original_read(database, statement, values)
if route == "attach" and statement.startswith("WITH observations AS"):
observe(database)
return result

def execute(database, statement, values=()):
result = original_execute(database, statement, values)
if route == "scratch" and statement.startswith("INSERT INTO lifecycle_state"):
observe(database)
return result

def directory(*args, **kwargs):
return original_directory(*args, dir=scratch_parent, **kwargs)

with ExitStack() as stack:
stack.enter_context(patch.object(_AttachedSources, "read", read))
stack.enter_context(patch.object(scratch._ScratchSources, "execute", execute))
stack.enter_context(patch.object(scratch.tempfile, "TemporaryDirectory", directory))
if route == "scratch":
stack.enter_context(patch.object(sql, "_ATTACHED_OBSERVATION_LIMIT", 0))
yield


def read_query(root, route):
with SnapshotAnalyzer(root) as analyzer:
return analyzer.execute_query(
"block",
QUERY_PARAMS,
exact_total=True,
query_limits=QueryLimits(
max_scratch_db_bytes=0 if route == "attach" else 512 * 1024**2
),
)


def query_worker(root, route, output):
try:
output.put(("result", read_query(root, route)))
except BaseException:
output.put(("error", traceback.format_exc()))


def process_reader(root, route, scratch_parent, ready, release, output):
def reached(evidence):
output.put(("stage", evidence))
ready.wait(WAIT_SECONDS)
if not release.wait(WAIT_SECONDS):
raise TimeoutError("Parent did not release the query-stage barrier")

with lifecycle_stage(route, scratch_parent, reached):
query_worker(root, route, output)


def cli_at_scratch_stage(root, scratch_parent, channel, stdout_path, stderr_path):
"""Call the real console entry point with Python's original signal behavior."""
from pt_snap_cli.cli import _safe_call

def reached(evidence):
channel.send(evidence)
# Parent either permits completion or sends an actual operating-system
# signal. No test-created handler turns SIGTERM into a Python exception.
if not channel.poll(WAIT_SECONDS):
raise TimeoutError("Parent did not release or signal the CLI")
assert channel.recv() == "continue"

sys.argv = [
"pt-snap",
"query",
str(root),
"--template-use",
"block",
"--params",
'{"limit":5}',
"--json",
]
with (
Path(stdout_path).open("w") as stdout,
Path(stderr_path).open("w") as stderr,
redirect_stdout(stdout),
redirect_stderr(stderr),
lifecycle_stage("scratch", scratch_parent, reached),
):
try:
raise SystemExit(_safe_call())
finally:
channel.close()
Loading
Loading