diff --git a/docs/en/dataset-acceptance.md b/docs/en/dataset-acceptance.md index aba5a24..4ca8707 100644 --- a/docs/en/dataset-acceptance.md +++ b/docs/en/dataset-acceptance.md @@ -169,3 +169,45 @@ on inspectable test-owned input. It does not execute live PyTorch/NPU collection the upstream msinsight producer, or its GUI. Those acceptance steps remain separate. No new committed pickle is introduced. The provenance gate runs before collection; the test-owned temporary input is written during the test. + +## Concurrent readers and process interruption + +`tests/core/test_dataset_concurrency.py` runs two and four independent readers +in both threads and spawned processes, against the same synthetic dataset and +through both ATTACH and scratch routes. Barriers follow an actual SQLite read +or scratch insert; all readers hold their own query resources concurrently. +Their complete results must agree, scratch directories must be distinct and +removed on normal completion, and source hashes must stay unchanged. This does +not promise that one analyzer or its SQLite connections can be shared by threads. + +`tests/test_dataset_signal_boundaries.py` sends actual POSIX signals to the CLI +after a completed scratch insert. Normal completion and SIGINT unwind product +resources and remove scratch. **Current SIGTERM and SIGKILL terminate the process +without that cleanup: the private `pt-snap-lifecycle-*` directory and `derived.db` +remain.** The tests record this limitation, rather than treating test teardown as +product cleanup or claiming issue #252's no-residue kill goal is satisfied. Source +hashes remain unchanged and no partial query output is published. No signal +handler, startup reclamation, or automatic deletion policy is introduced. + +These small deterministic tests run in the regular suite (signal cases are POSIX +only). Timeout/error cleanup is separate from abrupt termination: Python-level +unwinding is not evidence that SIGTERM or SIGKILL runs a `finally` block. + +`tests/test_dataset_mutation_acceptance.py` pauses a spawned CLI/API query after +its first shard has actually supplied rows or a count, then changes test-owned +members in place. Both the default strong and explicit immutable cache modes must +reject the changed generation without publishing partial rows/counts. Unchanged +runs succeed with independently expected results. Hash-work assertions quantify +compatibility-v1 cold/warm queries: default queries scan each manifest/member +three/two times respectively (admission plus the new final check). Explicit POSIX +immutable queries scan members once/zero times and the manifest twice/zero times. +The final check itself adds one full scan in default mode, zero in metadata-only +immutable mode, and shares the query's deadline. Non-POSIX immutable requests do +not reuse admission based on creation times: both cold and warm queries scan each +member twice and the manifest three times, including a full final hash guard. +Default strong warm reuse still works on non-POSIX systems. The tests exercise +this conservative policy through a resolver-local simulated non-POSIX case too; +that simulation is not evidence of actual Windows execution. These are work-count assertions, +not fixed wall-time or RSS promises. The guard detects these controlled mutations; +it does not create an atomic filesystem snapshot or prove all change-and-restore +races impossible. diff --git a/docs/en/querying.md b/docs/en/querying.md index d09cd04..6f9bc3f 100644 --- a/docs/en/querying.md +++ b/docs/en/querying.md @@ -193,6 +193,14 @@ check cancellation. Individual filesystem calls are not OS-level hard-preempted. `PT_SNAP_DATASET_CACHE=immutable` explicitly opts into metadata-only validation reuse only when the caller guarantees immutable finalized files and reliable filesystem change times; see [dataset validation and cache contracts](sharded-snapshotdb.md). +Every dataset query also verifies that admitted generation before returning rows or +counts, including count-only queries. A detected concurrent change fails the whole +query with a database-integrity error; no partial result is published. By default, +this final guard adds one complete manifest/member hash pass under the same +operation deadline. The explicit immutable mode retains its metadata-only guard +when its existing filesystem requirements hold. This is a change-detection boundary, +not an atomic filesystem snapshot: it does not prove all change-and-restore races +impossible or support querying mutable live SQLite artifacts. Source artifacts are never repaired or written. A large lifecycle query may create a private temporary derived database as described below. diff --git a/docs/zh/dataset-acceptance.md b/docs/zh/dataset-acceptance.md index e1ae70a..55b5a3a 100644 --- a/docs/zh/dataset-acceptance.md +++ b/docs/zh/dataset-acceptance.md @@ -134,3 +134,36 @@ PT_SNAP_ACCEPTANCE_WHEEL="$(realpath dist/.whl)" \ 新增导出门禁在可审查的测试输入上执行真实第一方 replay/export/query 链路;不执行 实时 PyTorch/NPU 采集、上游 msinsight 生产端或 GUI,这些验收仍独立。未增加任何提交 到仓库的 pickle。provenance 门禁在收集测试前运行;临时受信输入在测试期间自行写入。 + +## 并发读取与进程中断 + +`tests/core/test_dataset_concurrency.py` 对同一份合成数据集分别启动两个和四个 +独立读取者,覆盖线程、spawn 子进程,以及 ATTACH 和 scratch 两条路径。 +屏障位于实际 SQLite 读取或 scratch 插入完成之后,保证所有读取者同时持有各自的 +查询资源。完整结果必须一致,scratch 目录必须互不相同且在正常完成后移除,源文件 +哈希必须保持不变。这不代表同一个 analyzer 或其 SQLite 连接可以跨线程共享。 + +`tests/test_dataset_signal_boundaries.py` 在实际 scratch 插入完成后向 CLI 发送 +真实 POSIX 信号。正常完成和 SIGINT 会展开产品资源并移除 scratch。 +**目前 SIGTERM 和 SIGKILL 会直接终止进程,未执行上述清理,私有 +`pt-snap-lifecycle-*` 目录及 `derived.db` 会残留。** 测试明确记录这一限制, +不会把测试清理当作产品清理,也不宣称已经满足 issue #252 的 kill 后无残留目标。 +源文件哈希保持不变,不发布部分查询结果。本次未引入信号处理器、启动回收或自动删除策略。 + +这些小规模、确定性测试进入常规套件(信号用例仅限 POSIX)。超时及错误的清理与 +进程被直接终止是不同边界:Python 异常展开不能证明 SIGTERM 或 SIGKILL 会执行 +`finally`。 + +`tests/test_dataset_mutation_acceptance.py` 在 spawn 子进程中的 CLI/API 查询实际 +读取第一片的行或计数之后暂停,由父进程原地修改测试自有成员。默认强校验和显式 +immutable 缓存模式都必须拒绝已变更的 generation,不发布部分行或计数。 +未变更的运行必须成功且符合独立计算的结果。哈希工作量断言明确兼容 v1 的冷/热查询成本: +默认分别对每个 manifest/成员扫描三次/两次(准入加最终检查);POSIX immutable +模式对成员分别扫描一次/零次,对 manifest 分别扫描两次/零次。最终检查本身在默认模式 +增加一次完整扫描,在纯元数据 immutable 模式不增加哈希扫描,并共用查询 deadline。 +非 POSIX 上的 immutable 请求不会根据创建时间复用准入:冷/热查询都对每个成员扫描 +两次、对 manifest 扫描三次,包含完整的最终哈希检查。默认强校验在非 POSIX 上仍可复用 +经过内容验证的热准入。测试还通过仅替换 resolver 的平台对象,确定性覆盖这一保守策略; +此模拟不是实际 Windows 执行证据。 +这是工作量断言,不是固定耗时或 RSS 保证。检查会检测这些受控变更,但不会创建原子文件系统 +快照,也不证明所有先变更再恢复的竞态都能被检测。 diff --git a/docs/zh/querying.md b/docs/zh/querying.md index b791bec..5faa28f 100644 --- a/docs/zh/querying.md +++ b/docs/zh/querying.md @@ -163,8 +163,14 @@ SQLite、一个批次及单个 cell 有额外开销。默认仍执行强准入 分片数),新 CLI 进程仍冷启动。准入共用该 deadline:hash 分块、Python 行批次及 SQLite progress handler 都检查取消。 单个文件系统调用不提供 OS 级硬抢占。仅当调用方保证最终产物不可变且文件系统变更时间 可靠时,才可通过 `PT_SNAP_DATASET_CACHE=immutable` 显式启用纯元数据校验复用;参见 -[数据集校验与缓存合同](sharded-snapshotdb.md)。不修复或写入源产物;较大的生命周期查询 -可能生成下述私有临时派生数据库。 +[数据集校验与缓存合同](sharded-snapshotdb.md)。 +每次数据集查询在返回行或计数之前,都会再次检查已准入的 generation,纯计数查询也不例外。 +一旦检测到并发变更,整个查询以数据库完整性错误失败,不发布部分结果。默认情况下, +这一最终检查在同一操作 deadline 内额外扫描一次 manifest/全部成员的完整内容哈希。 +显式 immutable 模式在满足原有文件系统要求时,仍使用纯元数据的最终检查。 +这是变更检测边界,不是原子文件系统快照:不证明所有先变更再恢复的竞态都能被检测, +也不代表支持查询可变的在线 SQLite 产物。 +不修复或写入源产物;较大的生命周期查询可能生成下述私有临时派生数据库。 兼容导出器默认**每设备每片 500000 个真实事件**。该存储容量与 **100000 行 / 64 MiB 查询工作预算**独立:即使只有一个默认大小的分片,无足够筛选或分页的结果物化也可能 diff --git a/src/pt_snap_cli/core/query_service.py b/src/pt_snap_cli/core/query_service.py index bafec1b..83ad1c5 100644 --- a/src/pt_snap_cli/core/query_service.py +++ b/src/pt_snap_cli/core/query_service.py @@ -302,10 +302,10 @@ def execute_query( if isinstance(stack_bytes, int) and stack_bytes >= 0: summarize_dataset_stacks(result.rows, stack_bytes) budget.remaining() - if _resolved is not None: - # Inspection establishes the baseline. Reject changed contents - # after each composed step, before any report can be returned. - dataset.require_unchanged(budget) + # Inspection establishes the baseline for standalone queries and + # composed reports alike. Reject mixed-generation rows or counts + # before either adapter can publish any part of the result. + dataset.require_unchanged(budget) if result.scope is not None: result.scope["work_budget"] = budget.describe() return result diff --git a/tests/core/test_dataset_concurrency.py b/tests/core/test_dataset_concurrency.py new file mode 100644 index 0000000..8d742bd --- /dev/null +++ b/tests/core/test_dataset_concurrency.py @@ -0,0 +1,105 @@ +"""Independent readers overlap real query stages without sharing analyzers.""" + +import multiprocessing +import queue +import threading + +import pytest + +from tests.core.dataset_lifecycle_fixtures import build +from tests.dataset_concurrency_support import ( + ISOLATED_ENV, + WAIT_SECONDS, + lifecycle_stage, + process_reader, + query_worker, + read_query, +) +from tests.test_dataset_focus import hashes + + +@pytest.fixture(autouse=True) +def isolate(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + monkeypatch.setenv("HOME", str(tmp_path)) + for name in ISOLATED_ENV: + monkeypatch.delenv(name, raising=False) + + +@pytest.mark.parametrize("readers", [2, 4]) +@pytest.mark.parametrize("route", ["attach", "scratch"]) +@pytest.mark.parametrize("kind", ["threads", "processes"]) +def test_simultaneous_readers_return_complete_identical_results(tmp_path, kind, route, readers): + _, root, _, _, blocks = build(tmp_path, 7) + before = hashes(root) + scratch_parent = tmp_path / "scratch" + scratch_parent.mkdir() + with lifecycle_stage(route, scratch_parent, lambda evidence: None): + expected = read_query(root, route) + assert expected["returned"] == expected["total"] == len(blocks) + assert expected["total_is_exact"] and not expected["has_more"] + assert {row["id"] for row in expected["rows"]} == {block["id"] for block in blocks} + + context = multiprocessing.get_context("spawn") if kind == "processes" else threading + ready = context.Barrier(readers + 1) + release = context.Event() + output = context.Queue() if kind == "processes" else queue.Queue() + workers = [] + + def reached(evidence): + output.put(("stage", evidence)) + ready.wait(WAIT_SECONDS) + if not release.wait(WAIT_SECONDS): + raise TimeoutError("Parent did not release the query-stage barrier") + + try: + with lifecycle_stage(route, scratch_parent, reached): + for _ in range(readers): + worker = ( + context.Process( + target=process_reader, + args=(root, route, scratch_parent, ready, release, output), + ) + if kind == "processes" + else threading.Thread(target=query_worker, args=(root, route, output)) + ) + workers.append(worker) + worker.start() + # Every reader has completed a real SQL operation and holds its own + # connection open. No scheduling speed or arbitrary sleep is evidence. + ready.wait(WAIT_SECONDS) + stages = [output.get(timeout=WAIT_SECONDS) for _ in range(readers)] + assert all(kind == "stage" for kind, _ in stages), stages + identities = {(item["pid"], item["thread"]) for _, item in stages} + assert len(identities) == readers + if route == "scratch": + paths = {item["database"] for _, item in stages} + assert len(paths) == readers + assert all(item["inserted_rows"] > 0 for _, item in stages) + assert len(list(scratch_parent.glob("*/derived.db"))) == readers + else: + assert not list(scratch_parent.iterdir()) + assert hashes(root) == before + release.set() + results = [output.get(timeout=WAIT_SECONDS) for _ in range(readers)] + assert all(kind == "result" for kind, _ in results), results + assert all(result == expected for _, result in results) + finally: + release.set() + ready.abort() + for worker in workers: + worker.join(WAIT_SECONDS) + if kind == "processes" and worker.is_alive(): + worker.terminate() + worker.join(WAIT_SECONDS) + assert not worker.is_alive() + if kind == "processes": + assert worker.exitcode == 0 + worker.close() + if kind == "processes": + output.close() + output.join_thread() + # Only after all product contexts have exited do we check cleanup. The test + # does not remove scratch directories to manufacture this observation. + assert not list(scratch_parent.iterdir()) + assert hashes(root) == before diff --git a/tests/dataset_concurrency_support.py b/tests/dataset_concurrency_support.py new file mode 100644 index 0000000..5cbf969 --- /dev/null +++ b/tests/dataset_concurrency_support.py @@ -0,0 +1,143 @@ +"""Real query-stage barriers shared by thread, spawned-process and signal tests. + +Hooks observe a completed SQLite read/insert, then block on explicit IPC. They +never replace query results, install signal handlers, or perform product cleanup. +""" + +import os +import sys +import threading +import traceback +from contextlib import ExitStack, contextmanager, redirect_stderr, redirect_stdout +from pathlib import Path +from unittest.mock import patch + +from pt_snap_cli.api import QueryLimits, SnapshotAnalyzer +from pt_snap_cli.core import dataset_lifecycle_scratch as scratch +from pt_snap_cli.core import dataset_lifecycle_sql as sql +from pt_snap_cli.core.dataset_lifecycle_connection import _AttachedSources + +WAIT_SECONDS = 30 +QUERY_PARAMS = {"order_by": "id", "order_dir": "ASC", "limit": -1} +ISOLATED_ENV = ( + "PT_SNAP_DB_PATH", + "PT_SNAP_DATASET_CACHE", + "PT_SNAP_QUERY_TIMEOUT", + "PT_SNAP_MAX_WORK_ROWS", + "PT_SNAP_MAX_WORK_BYTES", + "PT_SNAP_MAX_SCRATCH_DB_BYTES", +) + + +@contextmanager +def lifecycle_stage(route, scratch_parent, reached): + original_read = _AttachedSources.read + original_execute = scratch._ScratchSources.execute + original_directory = scratch.tempfile.TemporaryDirectory + + def observe(database): + if getattr(database, "_test_stage_reached", False): + return + database._test_stage_reached = True + evidence = {"pid": os.getpid(), "thread": threading.get_ident(), "route": route} + if route == "scratch": + # These are actual product-created bytes, not a synthetic ready file. + cursor = database.connection.execute("PRAGMA database_list") + try: + evidence["database"] = next(row[2] for row in cursor if row[1] == "main") + finally: + cursor.close() + cursor = database.connection.execute("SELECT COUNT(*) FROM lifecycle_state") + try: + evidence["inserted_rows"] = cursor.fetchone()[0] + finally: + cursor.close() + reached(evidence) + + def read(database, statement, values=None): + result = original_read(database, statement, values) + if route == "attach" and statement.startswith("WITH observations AS"): + observe(database) + return result + + def execute(database, statement, values=()): + result = original_execute(database, statement, values) + if route == "scratch" and statement.startswith("INSERT INTO lifecycle_state"): + observe(database) + return result + + def directory(*args, **kwargs): + return original_directory(*args, dir=scratch_parent, **kwargs) + + with ExitStack() as stack: + stack.enter_context(patch.object(_AttachedSources, "read", read)) + stack.enter_context(patch.object(scratch._ScratchSources, "execute", execute)) + stack.enter_context(patch.object(scratch.tempfile, "TemporaryDirectory", directory)) + if route == "scratch": + stack.enter_context(patch.object(sql, "_ATTACHED_OBSERVATION_LIMIT", 0)) + yield + + +def read_query(root, route): + with SnapshotAnalyzer(root) as analyzer: + return analyzer.execute_query( + "block", + QUERY_PARAMS, + exact_total=True, + query_limits=QueryLimits( + max_scratch_db_bytes=0 if route == "attach" else 512 * 1024**2 + ), + ) + + +def query_worker(root, route, output): + try: + output.put(("result", read_query(root, route))) + except BaseException: + output.put(("error", traceback.format_exc())) + + +def process_reader(root, route, scratch_parent, ready, release, output): + def reached(evidence): + output.put(("stage", evidence)) + ready.wait(WAIT_SECONDS) + if not release.wait(WAIT_SECONDS): + raise TimeoutError("Parent did not release the query-stage barrier") + + with lifecycle_stage(route, scratch_parent, reached): + query_worker(root, route, output) + + +def cli_at_scratch_stage(root, scratch_parent, channel, stdout_path, stderr_path): + """Call the real console entry point with Python's original signal behavior.""" + from pt_snap_cli.cli import _safe_call + + def reached(evidence): + channel.send(evidence) + # Parent either permits completion or sends an actual operating-system + # signal. No test-created handler turns SIGTERM into a Python exception. + if not channel.poll(WAIT_SECONDS): + raise TimeoutError("Parent did not release or signal the CLI") + assert channel.recv() == "continue" + + sys.argv = [ + "pt-snap", + "query", + str(root), + "--template-use", + "block", + "--params", + '{"limit":5}', + "--json", + ] + with ( + Path(stdout_path).open("w") as stdout, + Path(stderr_path).open("w") as stderr, + redirect_stdout(stdout), + redirect_stderr(stderr), + lifecycle_stage("scratch", scratch_parent, reached), + ): + try: + raise SystemExit(_safe_call()) + finally: + channel.close() diff --git a/tests/test_dataset_mutation_acceptance.py b/tests/test_dataset_mutation_acceptance.py new file mode 100644 index 0000000..40b0258 --- /dev/null +++ b/tests/test_dataset_mutation_acceptance.py @@ -0,0 +1,248 @@ +"""A query may not publish rows/counts assembled from different dataset generations.""" + +import io +import json +import multiprocessing +import os +import sqlite3 +import sys +from collections import Counter +from contextlib import closing, redirect_stderr, redirect_stdout +from types import SimpleNamespace +from unittest.mock import patch + +import pytest + +from pt_snap_cli.api import SnapshotAnalyzer +from pt_snap_cli.core import dataset_resolver as resolver +from pt_snap_cli.core.dataset_sources import DatasetSourceResolver +from pt_snap_cli.core.errors import QueryTimeoutError +from tests.core.dataset_lifecycle_fixtures import build +from tests.dataset_concurrency_support import ISOLATED_ENV, WAIT_SECONDS +from tests.test_dataset_focus import hashes + + +def params_for(shape): + if shape == "count-only": + return {"action": 4, "limit": 0} + return {"order_by": "id", "order_dir": "ASC", "limit": -1} + + +def paused_query(root, adapter, shape, channel): + """Observe the first completed shard operation in an actual CLI/API query.""" + original = DatasetSourceResolver.read + observed = False + + def read(sources, path, statement, values=None): + nonlocal observed + result = original(sources, path, statement, values) + selected = ( + statement.startswith("SELECT MIN(id)") + if shape == "count-only" + else " WHERE t.id IN (" in statement + ) + if selected and not observed: + observed = True + channel.send({"stage": "first-shard-read", "member": path.name, "rows": len(result)}) + if not channel.poll(WAIT_SECONDS): + raise TimeoutError("Parent did not release the mutation barrier") + assert channel.recv() == "continue" + return result + + try: + with patch.object(DatasetSourceResolver, "read", read): + if adapter == "api": + try: + with SnapshotAnalyzer(root) as analyzer: + result = analyzer.execute_query( + "event", params_for(shape), exact_total=True + ) + outcome = {"status": "success", "payload": result} + except Exception as exc: + outcome = {"status": "error", "type": type(exc).__name__, "message": str(exc)} + else: + from pt_snap_cli.cli import _safe_call + + sys.argv = [ + "pt-snap", + "query", + str(root), + "--template-use", + "event", + "--params", + json.dumps(params_for(shape)), + "--exact-total", + "--json", + ] + stdout, stderr = io.StringIO(), io.StringIO() + with redirect_stdout(stdout), redirect_stderr(stderr): + code = _safe_call() + outcome = {"code": code, "stdout": stdout.getvalue(), "stderr": stderr.getvalue()} + channel.send(outcome) + finally: + channel.close() + + +@pytest.fixture(autouse=True) +def isolate(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + monkeypatch.setenv("HOME", str(tmp_path)) + for name in ISOLATED_ENV: + monkeypatch.delenv(name, raising=False) + + +@pytest.mark.parametrize("cache", ["strong", "immutable"]) +@pytest.mark.parametrize("adapter", ["api", "cli-json"]) +@pytest.mark.parametrize("shape", ["rows", "count-only"]) +@pytest.mark.parametrize("mutate", [False, True], ids=["unchanged", "mutated"]) +def test_query_rejects_mid_read_mutation_without_partial_payload( + tmp_path, monkeypatch, cache, adapter, shape, mutate +): + if cache == "immutable": + monkeypatch.setenv("PT_SNAP_DATASET_CACHE", "immutable") + _, root, _, trace, _ = build(tmp_path, 7) + before = hashes(root) + context = multiprocessing.get_context("spawn") + parent, child = context.Pipe() + process = context.Process(target=paused_query, args=(root, adapter, shape, child)) + process.start() + child.close() + try: + assert parent.poll(WAIT_SECONDS), "Query did not reach its first completed shard read" + stage = parent.recv() + assert stage == { + "stage": "first-shard-read", + "member": "slice_00000.db", + "rows": 1 if shape == "count-only" else 10, + } + if mutate: + # Test-owned artifacts only. The query is already admitted and has + # consumed real data; update every shard before it can read another. + statement = ( + "UPDATE trace_entry_0 SET action=7 WHERE id>=0" + if shape == "count-only" + else "UPDATE trace_entry_0 SET allocated=allocated+10000 WHERE id>=0" + ) + for member in sorted(root.glob("device_0/*.db")): + with closing(sqlite3.connect(member)) as connection, connection: + connection.execute(statement) + expected_hashes = hashes(root) + assert (expected_hashes != before) == mutate + parent.send("continue") + assert parent.poll(WAIT_SECONDS), "Query did not finish after barrier release" + outcome = parent.recv() + process.join(WAIT_SECONDS) + assert process.exitcode == 0 + if mutate: + if adapter == "api": + assert outcome == { + "status": "error", + "type": "DatabaseSchemaError", + "message": "Dataset members changed during operation; retry.", + } + else: + assert outcome["code"] == 1 and outcome["stdout"] == "" + payload = json.loads(outcome["stderr"]) + assert payload["error"]["code"] == "DATABASE_SCHEMA_INVALID" + assert ( + payload["error"]["message"] + == "Dataset members changed during operation; retry." + ) + assert not {"rows", "total", "returned", "scope"} & payload.keys() + else: + if adapter == "api": + assert outcome["status"] == "success" + payload = outcome["payload"] + else: + assert outcome["code"] == 0 and outcome["stderr"] == "" + payload = json.loads(outcome["stdout"]) + expected_total = ( + sum(event[1] == 4 for event in trace) if shape == "count-only" else len(trace) + ) + assert payload["total"] == expected_total and payload["total_is_exact"] + assert payload["returned"] == (0 if shape == "count-only" else len(trace)) + if shape == "rows": + assert [(row["id"], row["allocated"]) for row in payload["rows"]] == [ + (event[0], event[5]) for event in trace + ] + # Readers must not modify either the pristine or deliberately changed bytes. + assert hashes(root) == expected_hashes + finally: + parent.close() + if process.is_alive(): + process.kill() + process.join(WAIT_SECONDS) + process.close() + + +@pytest.mark.parametrize("cache", ["strong", "immutable"]) +@pytest.mark.parametrize("temperature", ["cold", "warm"]) +@pytest.mark.parametrize("platform", ["native", "simulated-non-posix"]) +def test_final_query_guard_hash_work_is_explicit( + tmp_path, monkeypatch, cache, temperature, platform, record_property +): + if cache == "immutable": + monkeypatch.setenv("PT_SNAP_DATASET_CACHE", "immutable") + if platform == "simulated-non-posix": + # Exercise the actual conservative policy on every CI OS without + # changing pathlib's host platform or weakening any source guard. + monkeypatch.setattr(resolver, "os", SimpleNamespace(name="nt", environ=os.environ)) + metadata_only = cache == "immutable" and resolver.os.name == "posix" + _, root, _, _, _ = build(tmp_path, 7) + counts, byte_counts = Counter(), Counter() + original = resolver._hash_file + + def tracked(path, **kwargs): + counts[path] += 1 + byte_counts[path] += path.stat().st_size + return original(path, **kwargs) + + if temperature == "warm": + resolver.DatasetResolver().inspect(root) + with SnapshotAnalyzer(root) as analyzer: + monkeypatch.setattr(resolver, "_hash_file", tracked) + result = analyzer.execute_query("event", {"limit": 0}, exact_total=True) + assert result["total"] == 80 + files = [root / "manifest.json", *root.glob("device_0/*.db")] + # Default strong reuse is content-verified on every OS. Only explicit + # immutable metadata reuse requires POSIX change times. Otherwise immutable + # requests repeat cold admission AND the final guard hashes all bytes. + if cache == "strong": + member_scans = manifest_scans = 3 if temperature == "cold" else 2 + elif metadata_only: + member_scans = int(temperature == "cold") + manifest_scans = 2 * member_scans + else: + member_scans, manifest_scans = 2, 3 + assert root not in resolver._VALIDATED + expected_counts = dict.fromkeys(files, member_scans) if member_scans else {} + if manifest_scans: + expected_counts[root / "manifest.json"] = manifest_scans + assert dict(counts) == expected_counts + assert sum(byte_counts.values()) == sum( + path.stat().st_size * count for path, count in expected_counts.items() + ) + record_property("query_hash_calls", sum(counts.values())) + record_property("query_hash_bytes", sum(byte_counts.values())) + record_property("final_guard_extra_scans", int(not metadata_only)) + + +def test_final_query_guard_uses_same_deadline_and_can_time_out(tmp_path, monkeypatch): + _, root, _, _, _ = build(tmp_path, 7) + original = resolver.ResolvedDataset.require_unchanged + read = DatasetSourceResolver.read + budgets = [] + + def tracked(sources, *args, **kwargs): + budgets.append(sources.budget) + return read(sources, *args, **kwargs) + + def expire(dataset, budget=None): + assert budgets and all(item is budget for item in budgets) + budget.started -= budget.timeout_s + 1 + return original(dataset, budget) + + monkeypatch.setattr(DatasetSourceResolver, "read", tracked) + monkeypatch.setattr(resolver.ResolvedDataset, "require_unchanged", expire) + with SnapshotAnalyzer(root) as analyzer, pytest.raises(QueryTimeoutError): + analyzer.execute_query("event", {"limit": 0}, exact_total=True, timeout_s=30) diff --git a/tests/test_dataset_signal_boundaries.py b/tests/test_dataset_signal_boundaries.py new file mode 100644 index 0000000..35411b8 --- /dev/null +++ b/tests/test_dataset_signal_boundaries.py @@ -0,0 +1,75 @@ +"""Characterize current CLI signal boundaries; no SIGTERM/SIGKILL cleanup promise.""" + +import json +import multiprocessing +import os +import signal +from pathlib import Path + +import pytest + +from tests.core.dataset_lifecycle_fixtures import build +from tests.dataset_concurrency_support import ISOLATED_ENV, WAIT_SECONDS, cli_at_scratch_stage +from tests.test_dataset_focus import hashes + +pytestmark = pytest.mark.skipif(os.name != "posix", reason="Requires actual POSIX signal semantics") + + +@pytest.mark.parametrize("action", ["complete", "SIGINT", "SIGTERM", "SIGKILL"]) +def test_cli_scratch_signal_boundary(tmp_path, monkeypatch, action): + monkeypatch.chdir(tmp_path) + monkeypatch.setenv("HOME", str(tmp_path)) + for name in ISOLATED_ENV: + monkeypatch.delenv(name, raising=False) + _, root, _, _, _ = build(tmp_path, 7) + before = hashes(root) + scratch_parent = tmp_path / "scratch" + scratch_parent.mkdir() + stdout, stderr = tmp_path / "stdout", tmp_path / "stderr" + context = multiprocessing.get_context("spawn") + parent, child = context.Pipe() + process = context.Process( + target=cli_at_scratch_stage, + args=(root, scratch_parent, child, stdout, stderr), + ) + process.start() + child.close() + try: + assert parent.poll(WAIT_SECONDS), "CLI did not reach a completed scratch insert" + stage = parent.recv() + derived = Path(stage["database"]) + assert stage["pid"] == process.pid + assert stage["inserted_rows"] > 0 and derived.stat().st_size > 0 + assert derived.parent.parent == scratch_parent + assert process.is_alive() + assert hashes(root) == before + if action == "complete": + parent.send("continue") + else: + os.kill(process.pid, getattr(signal, action)) + process.join(WAIT_SECONDS) + assert not process.is_alive(), "CLI did not exit after release/signal" + if action == "complete": + assert process.exitcode == 0 + assert json.loads(stdout.read_text())["returned"] == 5 + assert stderr.read_text() == "" + assert not list(scratch_parent.iterdir()) + elif action == "SIGINT": + assert process.exitcode in (1, 130) + assert stdout.read_text() == "" + assert json.loads(stderr.read_text())["error"]["message"] == "Aborted!" + assert not list(scratch_parent.iterdir()) + else: + # Existing Python/OS behavior terminates before finally/TemporaryDirectory + # cleanup. Residue is deliberately observed, not deleted as evidence. + assert process.exitcode == -getattr(signal, action) + assert stdout.read_text() == stderr.read_text() == "" + assert derived.is_file() + assert list(scratch_parent.iterdir()) == [derived.parent] + assert hashes(root) == before + finally: + parent.close() + if process.is_alive(): + process.kill() + process.join(WAIT_SECONDS) + process.close()