diff --git a/README.md b/README.md index 751670c..afacb19 100644 --- a/README.md +++ b/README.md @@ -71,6 +71,9 @@ canonical stack statistics, deduplicated lifecycles, terminal leak candidates an `report peak-memory` share bounded core semantics; custom SQL/overrides remain unsupported on datasets. See the [global support matrix](docs/en/querying.md#dataset-global-built-in-support-p3) and [dataset attribution and coverage](docs/en/querying.md#dataset-point-event-attribution). +Lifecycle queries exceeding SQLite's attachment limit use a private, page-limited +temporary derived database; source shards stay read-only. See the global support +matrix for storage limits, cleanup and memory caveats. Explicit `pt-snap import snapshot.pkl --format msinsight --json` produces `snapshot.pkl.msinsight/` for the fixed msinsight revision's **same original diff --git a/README_zh.md b/README_zh.md index 66abca3..1681c1e 100644 --- a/README_zh.md +++ b/README_zh.md @@ -62,6 +62,8 @@ pt-snap split snapshot.pkl --slices 4 --output snapshot-slices 终片泄漏候选及 `report peak-memory` 共用有界 core 语义;数据集仍不支持自定义 SQL/override。 详见[全局支持矩阵](docs/zh/querying.md#数据集全局内建支持p3)及 [数据集定点归因与覆盖](docs/zh/querying.md#数据集定点事件归因)。 +超过 SQLite 附加上限的生命周期查询使用私有、限制主库页数空间的临时派生数据库,源分片 +保持只读;存储限额、清理及内存边界见全局支持矩阵。 显式 `pt-snap import snapshot.pkl --format msinsight --json` 生成 `snapshot.pkl.msinsight/`,对齐固定 msinsight 版本的**同一原始 pickle + 邻接缓存**入口。 diff --git a/docs/en/querying.md b/docs/en/querying.md index 3d68d68..57d9383 100644 --- a/docs/en/querying.md +++ b/docs/en/querying.md @@ -161,7 +161,8 @@ check cancellation. Individual filesystem calls are not OS-level hard-preempted. `PT_SNAP_DATASET_CACHE=immutable` explicitly opts into metadata-only validation reuse only when the caller guarantees immutable finalized files and reliable filesystem change times; see [dataset validation and cache contracts](sharded-snapshotdb.md). -No temporary merge DB, repair or artifact write. +Source artifacts are never repaired or written. A large lifecycle query may create a +private temporary derived database as described below. The compatible exporter defaults to **500000 real events per device shard**. That storage capacity is separate from the **100000-row / 64-MiB query work @@ -180,16 +181,32 @@ formatted text is not their identity. Block/leak/lifetime reductions use read-on attached shards for proof, conflicts and latest observations before pagination; the attachment limit (normally ten) counts selected observation shards plus any other shards owning their allocation/free references. Slice queries discover those -proof owners before filtering or paging; exceeding the limit retains the budgeted -source-reading path. SQLite internal scans and temporary space are not bounded by -the fetched-row/byte budget. Summary and page (or lifetime buckets) each evaluate -the full lifecycle CTE, sharing the same deadline. No intermediate merged database -is created. +proof owners before filtering or paging. Above the limit, a query-private temporary +SQLite database retains only distinct event references and one narrow state row per +lifecycle. Sources are attached read-only one at a time: reference ownership/actions +are proved first, then all observations are reduced and conflicts checked before +filters, totals or pagination. Only returned candidates hydrate complete source events. +The small attached-set path is unchanged. + +The temporary directory is private (0700), its derived database is created with 0600 +permissions, and normal completion, errors and cancellation close connections and +remove the directory. A killed process or failed filesystem cleanup can leave private +residue; this is not secure erasure. Its main database page ceiling defaults to +512 MiB, configurable through the internal `QueryBudget.max_scratch_db_bytes` field from +4096 bytes to 4 GiB; zero disables this path and retains the original budgeted source +reader. This is not a new CLI option. Exceeding the page ceiling or running out of +storage rejects the entire operation. The ceiling does **not** bound rollback journals, +SQLite sorting files, total temporary space or RSS. Main/source page caches request +2 MiB each; these are cache targets, not memory guarantees. Internal SQL scans and +temporary work are not counted as Python-fetched rows. All stages, summary/page or +lifetime buckets and source hydration share the same diminishing deadline. A million-event dataset can therefore answer aggregates and small pages within the default work budget. Use `-n 5` or an explicit finite `limit` for large row-returning queries. Unlimited output, large offsets, many distinct stack groups, ordered-frame -reads or the attachment-limit fallback can still exceed the budget and fail honestly. +reads or a disabled temporary reducer can still exceed the fetched budget and fail +honestly. Many-lifecycle reductions can independently exceed the derived-page limit +or deadline. The limit counts rows fetched into Python and serialized output, not internal SQL scans, SQLite sorting space or total process memory. diff --git a/docs/zh/querying.md b/docs/zh/querying.md index 52616c8..5f390b8 100644 --- a/docs/zh/querying.md +++ b/docs/zh/querying.md @@ -136,7 +136,8 @@ SQLite、一个批次及单个 cell 有额外开销。默认仍执行强准入 分片数),新 CLI 进程仍冷启动。准入共用该 deadline:hash 分块、Python 行批次及 SQLite progress handler 都检查取消。 单个文件系统调用不提供 OS 级硬抢占。仅当调用方保证最终产物不可变且文件系统变更时间 可靠时,才可通过 `PT_SNAP_DATASET_CACHE=immutable` 显式启用纯元数据校验复用;参见 -[数据集校验与缓存合同](sharded-snapshotdb.md)。不生成合并临时 DB、不修复或写产物。 +[数据集校验与缓存合同](sharded-snapshotdb.md)。不修复或写入源产物;较大的生命周期查询 +可能生成下述私有临时派生数据库。 兼容导出器默认**每设备每片 500000 个真实事件**。该存储容量与 **100000 行 / 64 MiB 查询工作预算**独立:即使只有一个默认大小的分片,无足够筛选或分页的结果物化也可能 @@ -149,13 +150,23 @@ SQLite、一个批次及单个 cell 有额外开销。默认仍执行强准入 与总计数计算。规范化原始有序帧的身份不能用格式化文本替代,因此仍走受预算约束的来源 读取路径。block/leak/lifetime 通过只读附加分片先证明身份、检测冲突、选最新观察,再分页; 附加上限(通常十片)按选定观察分片与其分配/释放引用所属的其他分片合计。 -slice 查询在筛选、分页前发现所需的证明来源分片;超限时回退到原有受预算约束的来源 -读取路径。SQLite 内部扫描与临时空间不受取回行数/字节预算约束;汇总和分页(或存活时长 -分桶)各自执行完整的生命周期 CTE,共用同一 deadline。不创建中间合并数据库。 +slice 查询在筛选、分页前发现所需的证明来源分片。超限时使用查询专属的临时 SQLite +派生库,仅保留去重后的事件引用及每个生命周期的一条窄状态记录。源分片逐个只读挂载: +先证明引用的所属片及 action,再归约全部观察、检查冲突,之后才筛选、计数和分页。 +仅返回候选补全完整来源事件。未超过附加上限的原路径不变。 + +临时目录权限为 0700,派生文件以 0600 创建;正常完成、错误和取消均关闭连接并删除目录。 +强制杀进程或文件系统清理失败可能留下私有残余;这不是安全擦除。派生主库页数空间默认 +上限为 512 MiB,可通过内部 `QueryBudget.max_scratch_db_bytes` 字段设为 4096 字节至 +4 GiB;0 禁用该路径、沿用原有受预算约束的来源读取。这不是新增 CLI 选项。页数超限或 +磁盘满会拒绝整个查询。该上限**不包括**回滚 journal、SQLite 排序文件、全部临时空间或 +RSS。主库和源库各请求 2 MiB page cache,仅为缓存目标,不保证内存上限。SQL 内部扫描 +和临时计算不算 Python 取回行;全部阶段、汇总/分页或时长分桶及来源补全共用递减 deadline。 因此,百万事件数据集的聚合与小页面可在默认工作预算内完成。大量逐行结果请指定 `-n 5` -或有限 `limit`。无限制输出、大 offset、大量不同栈分组、有序帧读取及超过附加上限的回退 -路径仍可能超预算,并明确失败。预算统计进入 Python 的实际取回行和序列化输出,不限制 +或有限 `limit`。无限制输出、大 offset、大量不同栈分组、有序帧读取及禁用临时归约库后的 +来源读取仍可能超取回预算,并明确失败。大量生命周期归约也可能单独超出派生主库页数上限 +或 deadline。预算统计进入 Python 的实际取回行和序列化输出,不限制 SQL 内部扫描、SQLite 排序空间或整个进程内存。 ## 数据集定点事件归因 diff --git a/src/pt_snap_cli/core/dataset_global.py b/src/pt_snap_cli/core/dataset_global.py index 43ec456..cf03b6e 100644 --- a/src/pt_snap_cli/core/dataset_global.py +++ b/src/pt_snap_cli/core/dataset_global.py @@ -1,4 +1,4 @@ -"""Exact built-in global reductions, without a temporary merge database. +"""Exact built-in global reductions, with bounded fetched work. Shard SQL reduces counters, groups, and candidate windows before Python merging. Fetched work is cumulatively bounded by QueryBudget, including source hydration diff --git a/src/pt_snap_cli/core/dataset_lifecycle_connection.py b/src/pt_snap_cli/core/dataset_lifecycle_connection.py new file mode 100644 index 0000000..71beed0 --- /dev/null +++ b/src/pt_snap_cli/core/dataset_lifecycle_connection.py @@ -0,0 +1,88 @@ +"""Shared deadline-aware SQLite connection ownership for lifecycle reductions.""" + +from __future__ import annotations + +import math +import sqlite3 +import time + +from pt_snap_cli.core.dataset_contract import SliceRecord +from pt_snap_cli.core.dataset_files import require_readonly_member +from pt_snap_cli.core.dataset_sources import DatasetSourceResolver +from pt_snap_cli.core.errors import QueryExecutionError + + +class _AttachedSources: + def __init__( + self, + sources: DatasetSourceResolver, + items: list[SliceRecord] | None = None, + *, + database: str = ":memory:", + ) -> None: + self.sources = sources + self.items = sources.device.slices if items is None else items + remaining = sources.budget.remaining() + self.connection = sqlite3.connect( + database, uri=True, timeout=min(5.0, remaining) if remaining is not None else 5.0 + ) + self.connection.row_factory = sqlite3.Row + + def _configure_deadline(self): + budget = self.sources.budget + remaining = budget.remaining() + deadline = budget.started + budget.timeout_s if budget.timeout_s is not None else None + self.connection.set_progress_handler( + (lambda: int(time.monotonic() >= deadline)) if deadline is not None else None, 1000 + ) + # SQLite's default five-second lock wait must not outlive the operation. + wait_ms = math.ceil(1000 * min(5.0, remaining)) if remaining is not None else 5000 + self.connection.execute(f"PRAGMA busy_timeout={wait_ms}") + + def __enter__(self): + try: + self._configure_deadline() + self.connection.execute("PRAGMA trusted_schema=OFF") + for item in self.items: + self._configure_deadline() + path = self.sources.dataset.root / item.file + immutable = self.sources.dataset.callstack_layout == "v1" + require_readonly_member(self.sources.dataset.root, path, immutable=immutable) + uri = path.as_uri() + "?mode=ro" + ("&immutable=1" if immutable else "") + self.connection.execute(f'ATTACH DATABASE ? AS "s{item.index}"', [uri]) + self.connection.execute("PRAGMA query_only=ON") + self.sources.budget.remaining() + return self + except BaseException as exc: + self.connection.set_progress_handler(None, 0) + self.connection.close() + if isinstance(exc, sqlite3.Error): + self.sources.budget.remaining() + raise QueryExecutionError(str(exc)) from exc + raise + + def __exit__(self, exc_type, exc_value, traceback): + self.connection.set_progress_handler(None, 0) + self.connection.close() + return False + + def read(self, sql: str, values: list[object] | None = None): + budget = self.sources.budget + cursor = self.connection.cursor() + self.sources.query_count += 1 + try: + self._configure_deadline() + cursor.execute(sql, values or []) + result = [] + while batch := cursor.fetchmany(256): + rows = [dict(row) for row in batch] + budget.consume(rows) + result.extend(rows) + budget.remaining() + return result + except (sqlite3.Error, OverflowError) as exc: + budget.remaining() # Preserve the domain timeout on interruption. + raise QueryExecutionError(str(exc)) from exc + finally: + cursor.close() + self.connection.set_progress_handler(None, 0) diff --git a/src/pt_snap_cli/core/dataset_lifecycle_scratch.py b/src/pt_snap_cli/core/dataset_lifecycle_scratch.py new file mode 100644 index 0000000..e354af3 --- /dev/null +++ b/src/pt_snap_cli/core/dataset_lifecycle_scratch.py @@ -0,0 +1,191 @@ +"""Private disk-backed lifecycle reduction when the read-only ATTACH set is too large. + +Only references and one narrow row per identity are retained. The derived main +DB has a page ceiling; this does NOT limit all SQLite journals/sort files or RSS. +The fetched/output budget and operation deadline remain independent safeguards. +Source databases are never writable, even while the derived database is built. +""" + +from __future__ import annotations + +import os +import sqlite3 +import tempfile +from contextlib import contextmanager +from pathlib import Path + +from pt_snap_cli.core.dataset_files import require_readonly_member +from pt_snap_cli.core.dataset_lifecycle_connection import _AttachedSources +from pt_snap_cli.core.errors import QueryExecutionError + +_PAGE_SIZE = 4096 +_MAX_SCRATCH_BYTES = 4 * 1024 * 1024 * 1024 +_CACHE_KIB = 2048 +_COLUMNS = ("id", "address", "size", "requestedSize", "state", "allocEventId", "freeEventId") + + +class _ScratchSources(_AttachedSources): + def execute(self, sql, values=()): + self._configure_deadline() + self.sources.query_count += 1 + try: + self.connection.execute(sql, values).close() + self.sources.budget.remaining() + except (sqlite3.Error, OverflowError) as exc: + self.sources.budget.remaining() + if getattr(exc, "sqlite_errorcode", None) == getattr( + sqlite3, "SQLITE_FULL", 13 + ) or "database or disk is full" in str(exc): + raise QueryExecutionError( + "Dataset lifecycle scratch space exhausted (derived database page limit " + "or disk full); no partial global result." + ) from exc + raise QueryExecutionError(str(exc)) from exc + + @contextmanager + def source(self, item): + path = self.sources.dataset.root / item.file + immutable = self.sources.dataset.callstack_layout == "v1" + require_readonly_member(self.sources.dataset.root, path, immutable=immutable) + uri = path.as_uri() + "?mode=ro" + ("&immutable=1" if immutable else "") + self.execute("ATTACH DATABASE ? AS src", [uri]) + try: + # Bound each source's independent page cache too. URI mode=ro is the + # write barrier; query_only is connection-wide and would block main. + self.execute(f"PRAGMA src.cache_size=-{_CACHE_KIB}") + self.execute("PRAGMA src.mmap_size=0") + yield + finally: + # Detach must also work after deadline expiry or interruption. The + # outer owner closes the connection if detach itself fails. + self.connection.set_progress_handler(None, 0) + self.connection.execute("DETACH DATABASE src").close() + + def build(self, items, maximum): + self.connection.isolation_level = None + self.connection.set_authorizer( + lambda action, first, second, database, trigger: ( + sqlite3.SQLITE_DENY + if action in (sqlite3.SQLITE_INSERT, sqlite3.SQLITE_UPDATE, sqlite3.SQLITE_DELETE) + and database != "main" + else sqlite3.SQLITE_OK + ) + ) + self.execute("PRAGMA trusted_schema=OFF") + self.execute(f"PRAGMA main.page_size={_PAGE_SIZE}") + self.execute(f"PRAGMA main.max_page_count={maximum // _PAGE_SIZE}") + self.execute(f"PRAGMA main.cache_size=-{_CACHE_KIB}") + self.execute("PRAGMA main.mmap_size=0") + self.execute("PRAGMA temp_store=FILE") + self.execute("PRAGMA journal_mode=DELETE") + self.execute( + "CREATE TABLE refs(id INTEGER PRIMARY KEY, allocation_proved INTEGER NOT NULL " + "DEFAULT 0, completion_proved INTEGER NOT NULL DEFAULT 0)" + ) + device = self.sources.device.device_id + for item in items: + with self.source(item): + for column in ("allocEventId", "freeEventId"): + self.execute( + f'INSERT OR IGNORE INTO refs(id) SELECT "{column}" ' + f'FROM src."block_{device}" WHERE "{column}">=0' + ) + # Range ownership is authoritative. Never accept a same-ID event from a + # non-owner, a boundary/sentinel event, or an event with the wrong action. + for item in self.sources.device.slices: + needed = self.read( + "SELECT EXISTS(SELECT 1 FROM refs WHERE id BETWEEN ? AND ?) AS needed", + [item.start_event_id, item.end_event_id], + )[0]["needed"] + if needed: + with self.source(item): + self.execute( + "UPDATE refs SET allocation_proved=EXISTS(SELECT 1 FROM " + f'src."trace_entry_{device}" e WHERE e.id=refs.id AND e.id>=0 ' + "AND e.action=4), completion_proved=EXISTS(SELECT 1 FROM " + f'src."trace_entry_{device}" e WHERE e.id=refs.id AND e.id>=0 ' + "AND e.action=6) WHERE id BETWEEN ? AND ?", + [item.start_event_id, item.end_event_id], + ) + self.execute( + "CREATE TABLE lifecycle_state(token TEXT PRIMARY KEY COLLATE BINARY, " + + ", ".join(f'"{column}" INTEGER' for column in _COLUMNS) + + ", observation_slice_index INTEGER NOT NULL, allocation_proved INTEGER NOT NULL, " + "known_free INTEGER, invariant_conflict INTEGER NOT NULL DEFAULT 0, " + "free_conflict INTEGER NOT NULL DEFAULT 0) WITHOUT ROWID" + ) + latest = ( + "(excluded.observation_slice_index>lifecycle_state.observation_slice_index OR " + "(excluded.observation_slice_index=lifecycle_state.observation_slice_index " + "AND excluded.id>lifecycle_state.id))" + ) + updates = ", ".join( + f'"{column}"=CASE WHEN {latest} THEN excluded."{column}" ' + f'ELSE lifecycle_state."{column}" END' + for column in (*_COLUMNS, "observation_slice_index") + ) + conflict = " OR ".join( + f'lifecycle_state."{column}" IS NOT excluded."{column}"' + for column in ("address", "size", "requestedSize", "allocEventId") + ) + for item in items: + with self.source(item): + self.execute( + "INSERT INTO lifecycle_state(token, " + + ", ".join(f'"{column}"' for column in _COLUMNS) + + ", observation_slice_index, allocation_proved, known_free) " + "SELECT CASE WHEN a.allocation_proved THEN 'alloc:' || b.allocEventId " + "WHEN b.allocEventId=-1 AND b.id<0 THEN 'preexisting:' || b.id " + f"ELSE 'unproved:slice:{item.index}:block:' || b.id END, " + + ", ".join(f'b."{column}"' for column in _COLUMNS) + + f", {item.index}, COALESCE(a.allocation_proved,0), " + "CASE WHEN f.completion_proved THEN b.freeEventId END " + f'FROM src."block_{device}" b LEFT JOIN refs a ON a.id=b.allocEventId ' + "LEFT JOIN refs f ON f.id=b.freeEventId WHERE 1 " + "ON CONFLICT(token) DO UPDATE SET " + f"invariant_conflict=lifecycle_state.invariant_conflict OR ({conflict}), " + "free_conflict=lifecycle_state.free_conflict OR " + "(lifecycle_state.known_free IS NOT NULL AND excluded.known_free IS NOT NULL " + "AND lifecycle_state.known_free IS NOT excluded.known_free), " + "known_free=COALESCE(lifecycle_state.known_free,excluded.known_free), " + + updates + ) + self.execute("PRAGMA query_only=ON") + + +def _lifecycle_cte(terminal): + return f"""WITH latest AS ( + SELECT id,address,size,requestedSize,state,allocEventId, + COALESCE(known_free,freeEventId) AS freeEventId, + observation_slice_index,allocation_proved,token, + known_free IS NOT NULL AS completion_proved,invariant_conflict,free_conflict + FROM lifecycle_state + ), lifecycles AS ( + SELECT *, (observation_slice_index={terminal} AND state IN (0,1) + AND (freeEventId IS NULL OR freeEventId<0)) AS terminal_survivor + FROM latest + ) """ + + +@contextmanager +def scratch_lifecycles(sources, items): + maximum = sources.budget.max_scratch_db_bytes + if type(maximum) is not int or not _PAGE_SIZE <= maximum <= _MAX_SCRATCH_BYTES: + raise QueryExecutionError( + "Dataset lifecycle scratch limit must be 0 (disabled) or between 4096 bytes " + "and 4 GiB; this bounds derived database pages, not total temporary space or RSS." + ) + try: + with tempfile.TemporaryDirectory(prefix="pt-snap-lifecycle-") as directory: + path = Path(directory) / "derived.db" + descriptor = os.open(path, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600) + os.close(descriptor) + database = _ScratchSources(sources, [], database=str(path)) + try: + database.build(items, maximum) + yield database, _lifecycle_cte(sources.device.slices[-1].index) + finally: + database.__exit__(None, None, None) + except (OSError, sqlite3.Error) as exc: + sources.budget.remaining() + raise QueryExecutionError(f"Dataset lifecycle scratch storage failed: {exc}") from exc diff --git a/src/pt_snap_cli/core/dataset_lifecycle_sql.py b/src/pt_snap_cli/core/dataset_lifecycle_sql.py index 678b69e..48df7e7 100644 --- a/src/pt_snap_cli/core/dataset_lifecycle_sql.py +++ b/src/pt_snap_cli/core/dataset_lifecycle_sql.py @@ -4,96 +4,20 @@ latest-observation selection and page ranking; only aggregate/page rows cross the Python boundary. The fetched-row/byte budget does not bound SQLite's internal scans or temporary space. Summary and page/buckets each evaluate the full CTE; -all statements share the operation deadline. Larger required shard sets keep -the budgeted general implementation. +all statements share the operation deadline. Larger required shard sets use a +private, size-limited derived SQLite database; source members remain read-only. """ from __future__ import annotations -import math import sqlite3 -import time from contextlib import closing from typing import cast -from pt_snap_cli.core.dataset_contract import SliceRecord -from pt_snap_cli.core.dataset_files import require_readonly_member -from pt_snap_cli.core.dataset_sources import DatasetSourceResolver +from pt_snap_cli.core.dataset_lifecycle_connection import _AttachedSources from pt_snap_cli.core.errors import QueryExecutionError -class _AttachedSources: - def __init__( - self, sources: DatasetSourceResolver, items: list[SliceRecord] | None = None - ) -> None: - self.sources = sources - self.items = sources.device.slices if items is None else items - remaining = sources.budget.remaining() - self.connection = sqlite3.connect( - ":memory:", uri=True, timeout=min(5.0, remaining) if remaining is not None else 5.0 - ) - self.connection.row_factory = sqlite3.Row - - def _configure_deadline(self): - budget = self.sources.budget - remaining = budget.remaining() - deadline = budget.started + budget.timeout_s if budget.timeout_s is not None else None - self.connection.set_progress_handler( - (lambda: int(time.monotonic() >= deadline)) if deadline is not None else None, 1000 - ) - # SQLite's default five-second lock wait must not outlive the operation. - wait_ms = math.ceil(1000 * min(5.0, remaining)) if remaining is not None else 5000 - self.connection.execute(f"PRAGMA busy_timeout={wait_ms}") - - def __enter__(self): - try: - self._configure_deadline() - self.connection.execute("PRAGMA trusted_schema=OFF") - for item in self.items: - self._configure_deadline() - path = self.sources.dataset.root / item.file - immutable = self.sources.dataset.callstack_layout == "v1" - require_readonly_member(self.sources.dataset.root, path, immutable=immutable) - uri = path.as_uri() + "?mode=ro" + ("&immutable=1" if immutable else "") - self.connection.execute(f'ATTACH DATABASE ? AS "s{item.index}"', [uri]) - self.connection.execute("PRAGMA query_only=ON") - self.sources.budget.remaining() - return self - except BaseException as exc: - self.connection.set_progress_handler(None, 0) - self.connection.close() - if isinstance(exc, sqlite3.Error): - self.sources.budget.remaining() - raise QueryExecutionError(str(exc)) from exc - raise - - def __exit__(self, exc_type, exc_value, traceback): - self.connection.set_progress_handler(None, 0) - self.connection.close() - return False - - def read(self, sql: str, values: list[object] | None = None): - budget = self.sources.budget - cursor = self.connection.cursor() - self.sources.query_count += 1 - try: - self._configure_deadline() - cursor.execute(sql, values or []) - result = [] - while batch := cursor.fetchmany(256): - rows = [dict(row) for row in batch] - budget.consume(rows) - result.extend(rows) - budget.remaining() - return result - except (sqlite3.Error, OverflowError) as exc: - budget.remaining() # Preserve the domain timeout on interruption. - raise QueryExecutionError(str(exc)) from exc - finally: - cursor.close() - self.connection.set_progress_handler(None, 0) - - def _attachment_limit() -> int: # getlimit is unavailable on supported Python 3.10. Ten is SQLite's default. with closing(sqlite3.connect(":memory:")) as connection: @@ -213,11 +137,23 @@ def _filter(template, params): def lifecycle_query(sources, items, template, params, scope, candidate_limit): required = _required_slices(sources, items, _attachment_limit()) - if required is None: - return None predicate, values = _filter(template, params) - cte = _lifecycle_cte(sources, items, required) - with _AttachedSources(sources, required) as database: + if required is None: + from pt_snap_cli.core.dataset_lifecycle_scratch import scratch_lifecycles + + if sources.budget.max_scratch_db_bytes == 0 or _attachment_limit() < 1: + return None + execution = scratch_lifecycles(sources, items) + else: + from contextlib import contextmanager + + @contextmanager + def attached_lifecycles(): + with _AttachedSources(sources, required) as database: + yield database, _lifecycle_cte(sources, items, required) + + execution = attached_lifecycles() + with execution as (database, cte): summary = database.read( cte + f"""SELECT COUNT(*) AS lifecycles, COALESCE(SUM(allocation_proved),0) AS allocation_events_resolved, diff --git a/src/pt_snap_cli/core/dataset_sources.py b/src/pt_snap_cli/core/dataset_sources.py index 4907016..4dc49ec 100644 --- a/src/pt_snap_cli/core/dataset_sources.py +++ b/src/pt_snap_cli/core/dataset_sources.py @@ -68,6 +68,9 @@ class QueryBudget: max_work_bytes: int = 64 * 1024 * 1024 work_rows: int = 0 work_bytes: int = 0 + # Per-operation derived database pages, not total SQLite temp space or RSS. + # Zero retains the bounded in-Python fallback. Local/internal API only. + max_scratch_db_bytes: int = 512 * 1024 * 1024 def consume(self, rows: list[dict[str, object]]) -> None: """Bound cumulative fetched rows/serialized values, not merely output. diff --git a/tests/core/test_dataset_lifecycle_scratch.py b/tests/core/test_dataset_lifecycle_scratch.py new file mode 100644 index 0000000..c8902b1 --- /dev/null +++ b/tests/core/test_dataset_lifecycle_scratch.py @@ -0,0 +1,293 @@ +"""Many-owner lifecycle queries retain exact semantics and bounded private storage.""" + +import contextlib +import hashlib +import os +import sqlite3 +import time +from pathlib import Path + +import pytest + +from pt_snap_cli.core import dataset_lifecycle_scratch as scratch +from pt_snap_cli.core.dataset_resolver import DatasetResolver +from pt_snap_cli.core.dataset_sources import QueryBudget +from pt_snap_cli.core.errors import QueryExecutionError, QueryTimeoutError +from tests.core.test_dataset_sql_pushdown import run_resolved +from tests.test_dataset_focus import make_dataset + + +def many_owners(tmp_path): + root = make_dataset(tmp_path / "many", devices=(0,), slices=12) + with contextlib.closing(sqlite3.connect(root / "device_0/slice_00011.db")) as conn, conn: + conn.executemany( + "INSERT INTO block_0 VALUES (?,?,8,8,1,?,-1)", + ((i, 100 + i * 8, i) for i in range(24)), + ) + return root + + +def limited_connections(monkeypatch, limit): + if not hasattr(sqlite3.Connection, "setlimit"): + pytest.skip("Real per-connection SQLite limits require Python 3.11+") + original = sqlite3.connect + + def connect(*args, **kwargs): + connection = original(*args, **kwargs) + connection.setlimit(sqlite3.SQLITE_LIMIT_ATTACHED, limit) + return connection + + monkeypatch.setattr(sqlite3, "connect", connect) + + +def capture_storage(monkeypatch, tmp_path): + parent = tmp_path / "scratch" + parent.mkdir() + original = scratch.tempfile.TemporaryDirectory + created = [] + + def directory(*args, **kwargs): + result = original(*args, dir=parent, **kwargs) + created.append(Path(result.name)) + return result + + monkeypatch.setattr(scratch.tempfile, "TemporaryDirectory", directory) + return parent, created + + +@pytest.mark.parametrize("limit", [1, 2]) +@pytest.mark.parametrize("template", ["block", "leak_detection", "freed_block_lifetime"]) +@pytest.mark.parametrize("slice_index", [None, 11]) +def test_real_attach_limit_and_many_proof_owners( + tmp_path, monkeypatch, limit, template, slice_index +): + if template == "leak_detection" and slice_index is not None: + pytest.skip("Slice leak query is intentionally unsupported") + root = many_owners(tmp_path) + resolved = DatasetResolver().inspect(root) + expected = run_resolved( + resolved, + template, + slice_index=slice_index, + budget=QueryBudget(None, time.monotonic(), max_scratch_db_bytes=0), + ) + hashes = {p: hashlib.sha256(p.read_bytes()).hexdigest() for p in root.rglob("*.db")} + limited_connections(monkeypatch, limit) + parent, created = capture_storage(monkeypatch, tmp_path) + original = scratch._ScratchSources.build + observed = [] + + def build(database, items, maximum): + original(database, items, maximum) + path = Path(database.connection.execute("PRAGMA database_list").fetchone()[2]) + observed.append(database.connection) + assert os.stat(path.parent).st_mode & 0o777 == 0o700 + assert os.stat(path).st_mode & 0o777 == 0o600 + assert path.stat().st_size <= maximum + assert database.connection.getlimit(sqlite3.SQLITE_LIMIT_ATTACHED) == limit + + monkeypatch.setattr(scratch._ScratchSources, "build", build) + budget = QueryBudget(None, time.monotonic(), max_work_rows=128) + actual = run_resolved(resolved, template, slice_index=slice_index, budget=budget) + assert actual.rows == expected.rows + assert (actual.total, actual.has_more) == (expected.total, expected.has_more) + assert {k: v for k, v in actual.scope["source_coverage"].items() if k != "source_queries"} == { + k: v for k, v in expected.scope["source_coverage"].items() if k != "source_queries" + } + assert created and not list(parent.iterdir()) + assert all(not path.exists() for path in created) + for connection in observed: + with pytest.raises(sqlite3.ProgrammingError, match="closed"): + connection.execute("SELECT 1") + assert {p: hashlib.sha256(p.read_bytes()).hexdigest() for p in hashes} == hashes + assert not any(root.rglob("*-journal")) + assert budget.work_rows <= 128 + + +@pytest.mark.parametrize("maximum", [-1, 1, 4095, 4 * 1024**3 + 1, True, 1.5]) +def test_invalid_scratch_budget_is_domain_error(tmp_path, maximum): + resolved = DatasetResolver().inspect(many_owners(tmp_path)) + with pytest.raises(QueryExecutionError, match="scratch limit"): + run_resolved( + resolved, + "block", + budget=QueryBudget(None, time.monotonic(), max_scratch_db_bytes=maximum), + ) + + +def test_actual_page_limit_failure_cleans_private_storage(tmp_path, monkeypatch): + resolved = DatasetResolver().inspect(many_owners(tmp_path)) + parent, created = capture_storage(monkeypatch, tmp_path) + with pytest.raises(QueryExecutionError, match="scratch space exhausted"): + run_resolved( + resolved, "block", budget=QueryBudget(None, time.monotonic(), max_scratch_db_bytes=4096) + ) + assert created and not list(parent.iterdir()) + + +@pytest.mark.parametrize( + "phase", ["INSERT OR IGNORE", "UPDATE refs", "INSERT INTO lifecycle_state", "PRAGMA query_only"] +) +@pytest.mark.parametrize( + "failure", [QueryTimeoutError("expired"), KeyboardInterrupt(), OSError("full")] +) +def test_phase_failures_leave_no_storage_or_connection(tmp_path, monkeypatch, phase, failure): + resolved = DatasetResolver().inspect(many_owners(tmp_path)) + parent, created = capture_storage(monkeypatch, tmp_path) + original = scratch._ScratchSources.execute + connections = [] + + def execute(database, sql, values=()): + if sql.startswith(phase): + connections.append(database.connection) + raise failure + return original(database, sql, values) + + monkeypatch.setattr(scratch._ScratchSources, "execute", execute) + expected = QueryExecutionError if isinstance(failure, OSError) else type(failure) + with pytest.raises(expected): + run_resolved(resolved, "block") + assert created and not list(parent.iterdir()) + for connection in connections: + with pytest.raises(sqlite3.ProgrammingError, match="closed"): + connection.execute("SELECT 1") + + +def test_disabled_scratch_preserves_fetched_work_budget(tmp_path, monkeypatch): + resolved = DatasetResolver().inspect(many_owners(tmp_path)) + monkeypatch.setattr(scratch, "scratch_lifecycles", lambda *args: pytest.fail("disabled")) + with pytest.raises(QueryExecutionError, match="work budget"): + run_resolved( + resolved, + "block", + budget=QueryBudget(None, time.monotonic(), max_work_rows=3, max_scratch_db_bytes=0), + ) + + +def test_shared_deadline_expires_during_build_and_cleans_up(tmp_path, monkeypatch): + resolved = DatasetResolver().inspect(many_owners(tmp_path)) + parent, created = capture_storage(monkeypatch, tmp_path) + original = scratch._ScratchSources.execute + + def execute(database, sql, values=()): + if sql.startswith("UPDATE refs"): + database.sources.budget.started -= 10 + return original(database, sql, values) + + monkeypatch.setattr(scratch._ScratchSources, "execute", execute) + with pytest.raises(QueryTimeoutError): + run_resolved(resolved, "block", budget=QueryBudget(5, time.monotonic())) + assert created and not list(parent.iterdir()) + + +def test_scratch_source_writes_rejected_while_main_is_writable(tmp_path, monkeypatch): + resolved = DatasetResolver().inspect(many_owners(tmp_path)) + original = scratch._ScratchSources.source + visited = [] + + @contextlib.contextmanager + def source(database, item): + with original(database, item): + with pytest.raises(sqlite3.DatabaseError, match="authorized|readonly"): + database.connection.execute("DELETE FROM src.block_0") + visited.append(item.index) + yield + + monkeypatch.setattr(scratch._ScratchSources, "source", source) + run_resolved(resolved, "block") + assert len(set(visited)) == 12 + + +@pytest.mark.parametrize( + "variant", ["earlier_free", "null_invariant", "wrong_action", "same_slice_tie"] +) +@pytest.mark.parametrize("template", ["block", "leak_detection", "freed_block_lifetime"]) +def test_scratch_semantics_match_attached_path(tmp_path, monkeypatch, variant, template): + from tests.core.test_dataset_attribution import case + from tests.core.test_dataset_sql_pushdown import last_observation + + root = case(tmp_path, devices=(0,)) + # Mutate after admission to exercise reducer defenses independently of validation. + resolved = DatasetResolver().inspect(root) + if variant == "earlier_free": + with contextlib.closing(sqlite3.connect(last_observation(root, 0))) as conn, conn: + conn.execute("UPDATE block_0 SET state=0,freeEventId=-1 WHERE id=0") + elif variant == "null_invariant": + for path in root.glob("device_0/*.db"): + with contextlib.closing(sqlite3.connect(path)) as conn, conn: + conn.execute("UPDATE block_0 SET requestedSize=NULL WHERE id=0") + elif variant == "wrong_action": + for path in root.glob("device_0/*.db"): + with contextlib.closing(sqlite3.connect(path)) as conn, conn: + conn.execute("UPDATE trace_entry_0 SET action=5 WHERE id=0") + else: + with contextlib.closing(sqlite3.connect(last_observation(root, 0))) as conn, conn: + conn.execute( + "INSERT INTO block_0 SELECT 900,address,size,requestedSize,0,allocEventId,freeEventId " + "FROM block_0 WHERE id=0" + ) + params = {"offset": 1, "limit": 3} if template == "block" else {} + expected = run_resolved(resolved, template, params) + limited_connections(monkeypatch, 1) + actual = run_resolved(resolved, template, params) + assert actual.rows == expected.rows + assert (actual.total, actual.has_more) == (expected.total, expected.has_more) + + +@pytest.mark.parametrize("variant", ["invariant", "null", "free"]) +@pytest.mark.parametrize("params", [{"id": 7}, {"limit": 0}, {"offset": 100}]) +def test_scratch_conflicts_outside_page_still_fail(tmp_path, monkeypatch, variant, params): + from tests.core.test_dataset_attribution import case + from tests.core.test_dataset_sql_pushdown import last_observation + + root = case(tmp_path, devices=(0,)) + # Mutate after admission to exercise reducer defenses independently of validation. + resolved = DatasetResolver().inspect(root) + with contextlib.closing(sqlite3.connect(last_observation(root, 0))) as conn, conn: + if variant == "invariant": + conn.execute("UPDATE block_0 SET size=size+1 WHERE id=0") + elif variant == "null": + conn.execute("UPDATE block_0 SET requestedSize=NULL WHERE id=0") + else: + conn.execute("UPDATE block_0 SET freeEventId=4 WHERE id=0") + if variant == "free": + for path in root.glob("device_0/*.db"): + with contextlib.closing(sqlite3.connect(path)) as conn, conn: + conn.execute("UPDATE trace_entry_0 SET action=6 WHERE id=4") + with pytest.raises(QueryExecutionError, match="Conflicting"): + run_resolved(resolved, "block", params) + limited_connections(monkeypatch, 1) + with pytest.raises(QueryExecutionError, match="Conflicting"): + run_resolved(resolved, "block", params) + + +@pytest.mark.parametrize("stage", ["open", "detach"]) +def test_setup_or_detach_sqlite_failure_cleans_storage(tmp_path, monkeypatch, stage): + resolved = DatasetResolver().inspect(many_owners(tmp_path)) + parent, created = capture_storage(monkeypatch, tmp_path) + original = sqlite3.connect + connections = [] + + class DetachFailure(sqlite3.Connection): + def execute(self, sql, parameters=()): + if sql == "DETACH DATABASE src": + raise sqlite3.OperationalError("injected detach failure") + return super().execute(sql, parameters) + + def connect(path, *args, **kwargs): + if str(path).endswith("derived.db"): + if stage == "open": + raise sqlite3.OperationalError("injected open failure") + kwargs["factory"] = DetachFailure + connection = original(path, *args, **kwargs) + connections.append(connection) + return connection + return original(path, *args, **kwargs) + + monkeypatch.setattr(sqlite3, "connect", connect) + with pytest.raises(QueryExecutionError, match=f"injected {stage} failure"): + run_resolved(resolved, "block") + assert created and not list(parent.iterdir()) + for connection in connections: + with pytest.raises(sqlite3.ProgrammingError, match="closed"): + connection.execute("SELECT 1")