diff --git a/docs/en/querying.md b/docs/en/querying.md index 776a946..5d0f807 100644 --- a/docs/en/querying.md +++ b/docs/en/querying.md @@ -151,7 +151,7 @@ Context LRU. One diminishing deadline includes all shards, sources, grouping and totals; reports explicitly share **one** budget across peak selection and attribution. Cumulative fetched/output work is limited to **100000 rows / 64 MiB serialized values**, including repeated source reads. Exceeding either fails the entire operation, never an alleged -complete partial result. Materialization/sort are bounded by this work ceiling, **not** +complete partial result. Python materialization/sort are bounded by this work ceiling, **not** by max_rows. This is not a process RSS limit: Python/SQLite, one batch and individual cells have overhead. Strong manifest/member validation and hashing remain the default and share that deadline: hash chunks, Python row batches and SQLite progress handlers @@ -161,6 +161,29 @@ only when the caller guarantees immutable finalized files and reliable filesyste change times; see [dataset validation and cache contracts](sharded-snapshotdb.md). No temporary merge DB, repair or artifact write. +Global peak queries fetch scalar range evidence and at most three counter rows per +shard. Event/allocation pages apply SQL filters and ordering per shard, merge at most +`offset + limit` candidates per shard, and count matches separately. Range evidence +is measured before non-range filters. Ordinary text-stack statistics merge full +per-shard groups before global thresholds; averages use summed sizes and counts. +Canonical ordered-frame statistics retain the bounded source-reading path because +formatted text is not their identity. Block/leak/lifetime reductions use read-only +attached shards for proof, conflicts and latest observations before pagination; +the attachment limit (normally ten) counts selected observation shards plus any +other shards owning their allocation/free references. Slice queries discover those +proof owners before filtering or paging; exceeding the limit retains the budgeted +source-reading path. SQLite internal scans and temporary space are not bounded by +the fetched-row/byte budget. Summary and page (or lifetime buckets) each evaluate +the full lifecycle CTE, sharing the same deadline. No intermediate merged database +is created. + +A million-event dataset can therefore answer aggregates and small pages within the +default work budget. Use `-n 5` or an explicit finite `limit` for large row-returning +queries. Unlimited output, large offsets, many distinct stack groups, ordered-frame +reads or the attachment-limit fallback can still exceed the budget and fail honestly. +The limit counts rows fetched into Python and serialized output, not internal SQL +scans, SQLite sorting space or total process memory. + ## Dataset point-event attribution A complete compatibility-v1 or native-v2 directory/manifest supports `event` diff --git a/docs/zh/querying.md b/docs/zh/querying.md index 473d1e8..15964d7 100644 --- a/docs/zh/querying.md +++ b/docs/zh/querying.md @@ -130,13 +130,28 @@ exactness 标记诚实。`has_more/truncated` 描述窗口,不表示来源/范 数据集读取先分离批次再查外片来源,使用借用的有界 Context LRU。所有片、来源、分组、 精确总数共用递减 deadline;报告的峰值选择与归因明确共用**一个**预算。累计取回/输出工作上限为 **100000 行 / 64 MiB 序列化值**,含重复来源读取;超限整体失败,不返回冒充完整的部分 -结论。物化/排序由工作上限约束,**不由 max_rows 约束**;不是进程 RSS 上限,Python/ +结论。Python 物化/排序由工作上限约束,**不由 max_rows 约束**;不是进程 RSS 上限,Python/ SQLite、一个批次及单个 cell 有额外开销。默认仍对 manifest/成员执行完整校验和 hash, 并共用该 deadline:hash 分块、Python 行批次及 SQLite progress handler 都检查取消。 单个文件系统调用不提供 OS 级硬抢占。仅当调用方保证最终产物不可变且文件系统变更时间 可靠时,才可通过 `PT_SNAP_DATASET_CACHE=immutable` 显式启用进程内校验复用;参见 [数据集校验与缓存合同](sharded-snapshotdb.md)。不生成合并临时 DB、不修复或写产物。 +全局峰值查询每片只取范围汇总及最多三条计数器记录。event/allocation 在各片 SQL 中筛选、 +排序,每片最多取 `offset + limit` 个候选后全局归并,并单独统计匹配总数;实际范围证据在 +非范围筛选前计算。普通文本栈统计先合并各片完整分组,再应用全局阈值,平均值由总大小 +与总计数计算。规范化原始有序帧的身份不能用格式化文本替代,因此仍走受预算约束的来源 +读取路径。block/leak/lifetime 通过只读附加分片先证明身份、检测冲突、选最新观察,再分页; +附加上限(通常十片)按选定观察分片与其分配/释放引用所属的其他分片合计。 +slice 查询在筛选、分页前发现所需的证明来源分片;超限时回退到原有受预算约束的来源 +读取路径。SQLite 内部扫描与临时空间不受取回行数/字节预算约束;汇总和分页(或存活时长 +分桶)各自执行完整的生命周期 CTE,共用同一 deadline。不创建中间合并数据库。 + +因此,百万事件数据集的聚合与小页面可在默认工作预算内完成。大量逐行结果请指定 `-n 5` +或有限 `limit`。无限制输出、大 offset、大量不同栈分组、有序帧读取及超过附加上限的回退 +路径仍可能超预算,并明确失败。预算统计进入 Python 的实际取回行和序列化输出,不限制 +SQL 内部扫描、SQLite 排序空间或整个进程内存。 + ## 数据集定点事件归因 完整兼容 v1 或原生 v2 目录/manifest 支持 `event` 寻址,以及接受真实 `event_id` 的 diff --git a/src/pt_snap_cli/core/dataset_global.py b/src/pt_snap_cli/core/dataset_global.py index eea697f..43ec456 100644 --- a/src/pt_snap_cli/core/dataset_global.py +++ b/src/pt_snap_cli/core/dataset_global.py @@ -1,8 +1,9 @@ -"""Explicit built-in global algorithms; no local caps or temporary merge database. +"""Exact built-in global reductions, without a temporary merge database. -Fetched work is cumulatively bounded by QueryBudget, including sources and final -output. Sorting/materialization is O(bounded work), NOT O(max_rows). A budget -failure rejects the operation rather than returning an alleged global answer. +Shard SQL reduces counters, groups, and candidate windows before Python merging. +Fetched work is cumulatively bounded by QueryBudget, including source hydration +and final output. Unlimited/high-cardinality results can still exhaust the budget; +a failure rejects the operation rather than returning an alleged global answer. """ from __future__ import annotations @@ -11,6 +12,7 @@ from typing import cast from pt_snap_cli.core.dataset_contract import QueryScope, SliceRecord +from pt_snap_cli.core.dataset_lifecycle_sql import lifecycle_query from pt_snap_cli.core.dataset_sources import DatasetSourceResolver from pt_snap_cli.core.errors import InvalidParameterError, QueryExecutionError from pt_snap_cli.core.models import QueryResult @@ -68,44 +70,50 @@ def _scope( } -def _traces( - sources: DatasetSourceResolver, - low: int, - high: int, - items: list[SliceRecord], - columns: tuple[str, ...] = ("id", *METRICS), - *, - with_sources: bool = False, -): - projection = ", ".join(f't."{column}"' for column in columns) +def _range_statistics(sources, low, high, items, scope, predicate="1", values=None): + """Range evidence is deliberately collected BEFORE non-range filters.""" + count = matching = 0 + first = last = None for item in items: - trace = f'"trace_entry_{sources.device.device_id}"' - if with_sources: - if sources.dataset.callstack_layout == "v2": - sql = f"SELECT {projection}, t.callstackId, cs.callstack FROM {trace} t LEFT JOIN callstack cs ON t.callstackId=cs.id" - else: - sql = f"SELECT {projection}, t.callstack FROM {trace} t" - else: - sql = f"SELECT {projection} FROM {trace} t" - rows = sources.read( + row = sources.read( sources.dataset.root / item.file, - sql + " WHERE t.id>=0 AND t.id>=? AND t.id<=? ORDER BY t.id", - [low, high], - ) - yield item, rows + f"SELECT MIN(id) AS first, MAX(id) AS last, COUNT(*) AS count, " + f"COALESCE(SUM(CASE WHEN {predicate} THEN 1 ELSE 0 END), 0) AS matching " + f'FROM "trace_entry_{sources.device.device_id}" ' + "WHERE id>=0 AND id>=? AND id<=?", + [*(values or []), low, high], + )[0] + count += row["count"] + matching += row["matching"] + if row["first"] is not None: + first = row["first"] if first is None else min(first, row["first"]) + last = row["last"] if last is None else max(last, row["last"]) + scope["actual_range"] = { + "first_event_id": first, + "last_event_id": last, + "real_event_count": count, + } + return matching def _peaks(sources, low, high, items, template, scope): + _range_statistics(sources, low, high, items, scope) chosen: dict[str, dict[str, object]] = {} - count, first, last = 0, None, None - for _item, rows in _traces(sources, low, high, items, ("id", *METRICS)): - for row in rows: - sources.budget.remaining() - event = cast(int, row["id"]) - count += 1 - first = event if first is None else min(first, event) - last = event if last is None else max(last, event) - for metric in METRICS: + # These index-friendly queries fetch at most three narrow rows per shard. + # NULL counters never nominate an event, matching MAX + equality semantics. + for item in items: + for metric in METRICS: + rows = sources.read( + sources.dataset.root / item.file, + f"SELECT id, allocated, active, reserved " + f'FROM "trace_entry_{sources.device.device_id}" ' + f'WHERE id>=0 AND id>=? AND id<=? AND "{metric}" IS NOT NULL ' + f'ORDER BY "{metric}" DESC, id ASC LIMIT 1', + [low, high], + ) + if rows: + row = rows[0] + event = cast(int, row["id"]) if metric not in chosen or (cast(int, row[metric]), -event) > ( cast(int, chosen[metric][metric]), -cast(int, chosen[metric]["id"]), @@ -122,7 +130,9 @@ def _peaks(sources, low, high, items, template, scope): result[f"{other}_at_{metric}_peak"] = row.get(other) for other in ("active", "allocated"): result[f"reserved_{other}_gap_at_{metric}_peak"] = ( - cast(int, row["reserved"]) - cast(int, row[other]) if row else None + cast(int, row["reserved"]) - cast(int, row[other]) + if row and row["reserved"] is not None and row[other] is not None + else None ) if template == "allocator_gap": ids = [result[f"peak_{m}_event_id"] for m in METRICS] @@ -132,17 +142,31 @@ def _peaks(sources, low, high, items, template, scope): result[f"peak_{a}_event_id"] is not None and result[f"peak_{a}_event_id"] == result[f"peak_{b}_event_id"] ) - scope["actual_range"] = { - "first_event_id": first, - "last_event_id": last, - "real_event_count": count, - } scope["peak_owner_slices"] = { m: sources.owner(cast(int, r["id"])).index for m, r in chosen.items() } return [result] +def _predicates(params, fields, alias=""): + clauses, values = [], [] + for field in fields: + for prefix, operator in (("", "="), ("min_", ">="), ("max_", "<=")): + value = params.get(prefix + field) + if value is not None: + clauses.append(f'{alias}"{field}" {operator} ?') + values.append(value) + return " AND ".join(clauses) or "1", values + + +def _window(params, max_rows): + offset = max(0, cast(int, params.get("offset", 0))) + limit = cast(int, params.get("limit", -1)) + if max_rows is not None and max_rows > 0: + limit = min(limit, max_rows) if limit >= 0 else max_rows + return offset, limit + + def _matches(row: dict[str, object], params: dict[str, object], fields: tuple[str, ...]) -> bool: for field in fields: value = params.get(field) @@ -163,36 +187,45 @@ def _sort(rows: list[dict[str, object]], key: str, descending: bool) -> None: rows.sort(key=lambda r: (r[key] is not None, r[key]), reverse=descending) -def _events(sources, low, high, items, params, template, scope): +def _events(sources, low, high, items, params, template, scope, candidate_limit): + fields = METRICS + if template == "event": + fields += ("id", "action", "address", "size", "stream") + predicate, values = _predicates(params, fields) + total = _range_statistics(sources, low, high, items, scope, predicate, values) rows = [] - first, last, count = None, None, 0 + order = params["order_by"] + direction = params["order_dir"] columns = ("id", *METRICS) if template == "event": - columns += ("action", "address", "size", "stream") - with_sources = template == "event" and params["order_by"] == "callstack" - for _item, batch in _traces(sources, low, high, items, columns, with_sources=with_sources): - if with_sources: - sources.events_from_rows(_item, batch, with_frames=False) - for row in batch: - sources.budget.remaining() - event = cast(int, row["id"]) - first = event if first is None else min(first, event) - last = event if last is None else max(last, event) - count += 1 - fields = ("allocated", "active", "reserved") - if template == "event": - fields += ("id", "action", "address", "size", "stream") - if _matches(row, params, fields): - rows.append(row) - scope["actual_range"] = { - "first_event_id": first, - "last_event_id": last, - "real_event_count": count, - } - # callstack sorting already has joined text; resolving sources here would - # eagerly fetch ordered frames that only the returned page needs. - _sort(rows, params["order_by"], params["order_dir"] == "DESC") - return rows + columns = ("id", "action", "address", "size", "stream", *METRICS) + for item in items: + trace = f'"trace_entry_{sources.device.device_id}"' + projection = ", ".join(f't."{column}"' for column in columns) + join = "" + if order == "callstack": + if sources.dataset.callstack_layout == "v2": + projection += ", t.callstackId, cs.callstack" + join = " LEFT JOIN callstack cs ON t.callstackId=cs.id" + else: + projection += ", t.callstack" + row_predicate, row_values = _predicates(params, fields, "t.") + # Dataset text identity/order is exact, independent of source collation. + sort_column = '"callstack" COLLATE BINARY' if order == "callstack" else f'"{order}"' + batch = sources.read( + sources.dataset.root / item.file, + f"SELECT {projection} FROM {trace} t{join} " + f"WHERE t.id>=0 AND t.id>=? AND t.id<=? AND {row_predicate} " + f"ORDER BY {sort_column} {direction}, t.id {direction} LIMIT ?", + [low, high, *row_values, candidate_limit], + ) + if order == "callstack": + # Full candidate text has already crossed the budget boundary. Reuse + # its event data; ordered frames are needed only for the final page. + sources.events_from_rows(item, batch, with_frames=False) + rows.extend(batch) + _sort(rows, order, direction == "DESC") + return rows, total def _lifecycles(sources, items, scope): @@ -302,18 +335,103 @@ def _lifecycles(sources, items, scope): return rows +def _text_stacks(sources, low, high, items, params, scope): + """Merge shard sufficient statistics, never local HAVING/ranking windows.""" + grouped = {} + events = 0 + for item in items: + trace = f'"trace_entry_{sources.device.device_id}"' + if sources.dataset.callstack_layout == "v2": + table = f"{trace} t LEFT JOIN callstack cs ON t.callstackId=cs.id" + text, present = "cs.callstack", "t.callstackId IS NOT NULL" + else: + table, text, present = f"{trace} t", "t.callstack", "t.callstack IS NOT NULL" + batch = sources.read( + sources.dataset.root / item.file, + f"SELECT {text} AS callstack, COUNT(*) AS alloc_count, SUM(t.size) AS total_size, " + f"MAX(t.size) AS max_size, MIN(t.id) AS stack_event_id FROM {table} " + f"WHERE t.id>=0 AND t.id>=? AND t.id<=? AND {present} GROUP BY {text} COLLATE BINARY", + [low, high], + ) + for local in batch: + sources.budget.remaining() + value = local["callstack"] + identity = ( + f"{sources.namespace}:text:sha256:{hashlib.sha256(value.encode('utf-8')).hexdigest()}" + if isinstance(value, str) + else f"{sources.namespace}:category:missing" + ) + kind = "captured" if isinstance(value, str) and value != "" else "missing" + row = grouped.get(identity) + if row is None: + row = dict(local) + row.update( + { + "callstack": value if value is not None else "[missing callstack]", + "source_stack_id": identity, + "stack_kind": kind, + "text_kind": kind, + "frames_status": "text_only", + } + ) + grouped[identity] = row + else: + row["alloc_count"] += local["alloc_count"] + row["total_size"] += local["total_size"] + row["max_size"] = max(row["max_size"], local["max_size"]) + row["stack_event_id"] = min(row["stack_event_id"], local["stack_event_id"]) + events += local["alloc_count"] + return _finish_stacks(sources, grouped, params, scope, events, 0) + + +def _finish_stacks(sources, grouped, params, scope, events, ordered): + rows = [ + r + for r in grouped.values() + if r["alloc_count"] >= params["min_count"] and r["total_size"] >= params["min_size"] + ] + for row in rows: + row["avg_size"] = row["total_size"] / row["alloc_count"] + rows.sort( + key=lambda r: ( + -r["total_size"], + r["callstack"] or "", + r["stack_kind"] == "missing", + r["source_stack_id"], + ) + ) + scope["source_coverage"] = { + "range_complete": True, + "coverage_scope": "all_real_stack_bearing_actions_before_global_thresholds", + "stack_events": events, + "ordered_frame_events": ordered, + "source_queries": sources.query_count, + } + return rows + + def _stacks(sources, low, high, items, params, scope): + if sources.dataset.ordered_frames_version != 1: + return _text_stacks(sources, low, high, items, params, scope) grouped = {} events = ordered = 0 - for _item, batch in _traces( - sources, - low, - high, - items, - ("id", *METRICS, "action", "address", "size", "stream"), - with_sources=True, - ): - refs = sources.events_from_rows(_item, batch) + for item in items: + trace = f'"trace_entry_{sources.device.device_id}"' + columns = ("id", "action", "address", "size", "stream", *METRICS) + projection = ", ".join(f't."{column}"' for column in columns) + join = "" + if sources.dataset.callstack_layout == "v2": + projection += ", t.callstackId, cs.callstack" + join = " LEFT JOIN callstack cs ON t.callstackId=cs.id" + else: + projection += ", t.callstack" + batch = sources.read( + sources.dataset.root / item.file, + f"SELECT {projection} FROM {trace} t{join} " + "WHERE t.id>=0 AND t.id>=? AND t.id<=? ORDER BY t.id", + [low, high], + ) + refs = sources.events_from_rows(item, batch) for source in refs.values(): sources.budget.remaining() text = source.event.get("callstack") @@ -351,29 +469,7 @@ def _stacks(sources, low, high, items, params, scope): row["text_kind"] = "captured" events += 1 ordered += source.frames is not None - rows = [ - r - for r in grouped.values() - if r["alloc_count"] >= params["min_count"] and r["total_size"] >= params["min_size"] - ] - for row in rows: - row["avg_size"] = row["total_size"] / row["alloc_count"] - rows.sort( - key=lambda r: ( - -r["total_size"], - r["callstack"] or "", - r["stack_kind"] == "missing", - r["source_stack_id"], - ) - ) - scope["source_coverage"] = { - "range_complete": True, - "coverage_scope": "all_real_stack_bearing_actions_before_global_thresholds", - "stack_events": events, - "ordered_frame_events": ordered, - "source_queries": sources.query_count, - } - return rows + return _finish_stacks(sources, grouped, params, scope, events, ordered) def global_query( @@ -390,10 +486,15 @@ def global_query( ) low, high, items = _range(sources, range_params, slice_index) scope = _scope(sources, low, high, items) + offset, limit = _window(params, max_rows) + candidate_limit = offset + limit if limit >= 0 else -1 + total_override = None if template in ("memory_peak", "allocator_gap"): rows = _peaks(sources, low, high, items, template, scope) elif template in ("event", "allocation"): - rows = _events(sources, low, high, items, params, template, scope) + rows, total_override = _events( + sources, low, high, items, params, template, scope, candidate_limit + ) elif template == "callstack_analysis": rows = _stacks(sources, low, high, items, params, scope) elif template == "preexisting_live": @@ -411,56 +512,61 @@ def global_query( raise QueryExecutionError( "Dataset leak_detection requires terminal dataset scope, not --slice; select an explicit member for local observations." ) - rows = _lifecycles(sources, items, scope) - if template == "block": - rows = [ - r - for r in rows - if _matches( - r, - params, - ("id", "address", "size", "requestedSize", "allocEventId", "freeEventId"), - ) - ] - _sort(rows, cast(str, params["order_by"]), params["order_dir"] == "DESC") - elif template == "leak_detection": - rows = [ - r - for r in rows - if r["terminal_survivor"] - and r["allocation_source"] is not None - and cast(int, r["size"]) >= cast(int, params["min_size"]) - ] - _sort(rows, "size", True) - elif template == "freed_block_lifetime": - buckets = {} - for row in rows: - if row["allocation_source"] is None or row["free_source"] is None: - continue - distance = cast(int, row["freeEventId"]) - cast(int, row["allocEventId"]) - if distance < 0: - continue - index = next( - (i for i, cap in enumerate((1000, 5000, 20000, 100000)) if distance < cap), 4 - ) - bucket = buckets.setdefault( - index, - { - "lifetime_events": ("<1k", "1k-5k", "5k-20k", "20k-100k", ">=100k")[index], - "block_count": 0, - "size_bytes": 0, - }, - ) - bucket["block_count"] += 1 - bucket["size_bytes"] += row["size"] - rows = [buckets[i] for i in sorted(buckets)] + pushed = lifecycle_query(sources, items, template, params, scope, candidate_limit) + if pushed is not None: + rows, total_override = pushed else: - raise QueryExecutionError("No dataset-global merge implementation for this template.") - total = len(rows) - offset = max(0, cast(int, params.get("offset", 0))) - limit = cast(int, params.get("limit", -1)) - if max_rows is not None and max_rows > 0: - limit = min(limit, max_rows) if limit >= 0 else max_rows + rows = _lifecycles(sources, items, scope) + if template == "block": + rows = [ + r + for r in rows + if _matches( + r, + params, + ("id", "address", "size", "requestedSize", "allocEventId", "freeEventId"), + ) + ] + _sort(rows, cast(str, params["order_by"]), params["order_dir"] == "DESC") + elif template == "leak_detection": + rows = [ + r + for r in rows + if r["terminal_survivor"] + and r["allocation_source"] is not None + and cast(int, r["size"]) >= cast(int, params["min_size"]) + ] + _sort(rows, "size", True) + elif template == "freed_block_lifetime": + buckets = {} + for row in rows: + if row["allocation_source"] is None or row["free_source"] is None: + continue + distance = cast(int, row["freeEventId"]) - cast(int, row["allocEventId"]) + if distance < 0: + continue + index = next( + (i for i, cap in enumerate((1000, 5000, 20000, 100000)) if distance < cap), + 4, + ) + bucket = buckets.setdefault( + index, + { + "lifetime_events": ("<1k", "1k-5k", "5k-20k", "20k-100k", ">=100k")[ + index + ], + "block_count": 0, + "size_bytes": 0, + }, + ) + bucket["block_count"] += 1 + bucket["size_bytes"] += row["size"] + rows = [buckets[i] for i in sorted(buckets)] + else: + raise QueryExecutionError( + "No dataset-global merge implementation for this template." + ) + total = len(rows) if total_override is None else total_override selected = rows[offset : offset + limit] if limit >= 0 else rows[offset:] if template == "event": refs = sources.events(cast(int, r["id"]) for r in selected) diff --git a/src/pt_snap_cli/core/dataset_lifecycle_sql.py b/src/pt_snap_cli/core/dataset_lifecycle_sql.py new file mode 100644 index 0000000..678b69e --- /dev/null +++ b/src/pt_snap_cli/core/dataset_lifecycle_sql.py @@ -0,0 +1,319 @@ +"""Read-only relational lifecycle reduction over a bounded set of attached shards. + +No rows are copied to a merge database. SQLite performs proof, conflict checks, +latest-observation selection and page ranking; only aggregate/page rows cross the +Python boundary. The fetched-row/byte budget does not bound SQLite's internal +scans or temporary space. Summary and page/buckets each evaluate the full CTE; +all statements share the operation deadline. Larger required shard sets keep +the budgeted general implementation. +""" + +from __future__ import annotations + +import math +import sqlite3 +import time +from contextlib import closing +from typing import cast + +from pt_snap_cli.core.dataset_contract import SliceRecord +from pt_snap_cli.core.dataset_files import require_readonly_member +from pt_snap_cli.core.dataset_sources import DatasetSourceResolver +from pt_snap_cli.core.errors import QueryExecutionError + + +class _AttachedSources: + def __init__( + self, sources: DatasetSourceResolver, items: list[SliceRecord] | None = None + ) -> None: + self.sources = sources + self.items = sources.device.slices if items is None else items + remaining = sources.budget.remaining() + self.connection = sqlite3.connect( + ":memory:", uri=True, timeout=min(5.0, remaining) if remaining is not None else 5.0 + ) + self.connection.row_factory = sqlite3.Row + + def _configure_deadline(self): + budget = self.sources.budget + remaining = budget.remaining() + deadline = budget.started + budget.timeout_s if budget.timeout_s is not None else None + self.connection.set_progress_handler( + (lambda: int(time.monotonic() >= deadline)) if deadline is not None else None, 1000 + ) + # SQLite's default five-second lock wait must not outlive the operation. + wait_ms = math.ceil(1000 * min(5.0, remaining)) if remaining is not None else 5000 + self.connection.execute(f"PRAGMA busy_timeout={wait_ms}") + + def __enter__(self): + try: + self._configure_deadline() + self.connection.execute("PRAGMA trusted_schema=OFF") + for item in self.items: + self._configure_deadline() + path = self.sources.dataset.root / item.file + immutable = self.sources.dataset.callstack_layout == "v1" + require_readonly_member(self.sources.dataset.root, path, immutable=immutable) + uri = path.as_uri() + "?mode=ro" + ("&immutable=1" if immutable else "") + self.connection.execute(f'ATTACH DATABASE ? AS "s{item.index}"', [uri]) + self.connection.execute("PRAGMA query_only=ON") + self.sources.budget.remaining() + return self + except BaseException as exc: + self.connection.set_progress_handler(None, 0) + self.connection.close() + if isinstance(exc, sqlite3.Error): + self.sources.budget.remaining() + raise QueryExecutionError(str(exc)) from exc + raise + + def __exit__(self, exc_type, exc_value, traceback): + self.connection.set_progress_handler(None, 0) + self.connection.close() + return False + + def read(self, sql: str, values: list[object] | None = None): + budget = self.sources.budget + cursor = self.connection.cursor() + self.sources.query_count += 1 + try: + self._configure_deadline() + cursor.execute(sql, values or []) + result = [] + while batch := cursor.fetchmany(256): + rows = [dict(row) for row in batch] + budget.consume(rows) + result.extend(rows) + budget.remaining() + return result + except (sqlite3.Error, OverflowError) as exc: + budget.remaining() # Preserve the domain timeout on interruption. + raise QueryExecutionError(str(exc)) from exc + finally: + cursor.close() + self.connection.set_progress_handler(None, 0) + + +def _attachment_limit() -> int: + # getlimit is unavailable on supported Python 3.10. Ten is SQLite's default. + with closing(sqlite3.connect(":memory:")) as connection: + getter = getattr(connection, "getlimit", None) + category = getattr(sqlite3, "SQLITE_LIMIT_ATTACHED", None) + return getter(category) if getter is not None and category is not None else 10 + + +def _required_slices(sources, items, limit): + """Discover proof owners without fetching individual block/event references.""" + required = {item.index: item for item in items} + if len(required) > limit: + return None + if len(required) == len(sources.device.slices): + return list(required.values()) + for item in items: + # Both references may point outside the observation scope. Discover all + # owners before filtering/pagination so lifecycle conflicts remain visible. + queries = [] + for field in ("allocEventId", "freeEventId"): + cases = " ".join( + f'WHEN "{field}" BETWEEN {owner.start_event_id} AND {owner.end_event_id} ' + f"THEN {owner.index}" + for owner in sources.device.slices + if owner.index not in required + ) + queries.append( + f"SELECT CASE {cases} END AS owner_index " + f'FROM "block_{sources.device.device_id}"' + ) + rows = sources.read( + sources.dataset.root / item.file, + "SELECT owner_index FROM (" + " UNION ".join(queries) + ") " + "WHERE owner_index IS NOT NULL LIMIT ?", + [limit - len(required) + 1], + ) + for row in rows: + owner = sources.device.slices[row["owner_index"]] + required[owner.index] = owner + if len(required) > limit: + return None + if len(required) == len(sources.device.slices): + break + return sorted(required.values(), key=lambda item: item.index) + + +def _proof(sources, items, field, action): + device = sources.device.device_id + cases = " ".join( + f'WHEN b."{field}" BETWEEN {item.start_event_id} AND {item.end_event_id} ' + f'THEN EXISTS(SELECT 1 FROM "s{item.index}"."trace_entry_{device}" e ' + f'WHERE e.id=b."{field}" AND e.id>=0 AND e.action={action})' + for item in items + ) + return f"CASE {cases} ELSE 0 END" + + +def _lifecycle_cte(sources, items, proof_items): + allocated = _proof(sources, proof_items, "allocEventId", 4) + completed = _proof(sources, proof_items, "freeEventId", 6) + observations = " UNION ALL ".join( + f"SELECT b.*, {item.index} AS observation_slice_index, " + f"{allocated} AS allocation_proved, " + f"CASE WHEN {completed} THEN b.freeEventId END AS completed_free " + f'FROM "s{item.index}"."block_{sources.device.device_id}" b' + for item in items + ) + invariants = ("address", "size", "requestedSize", "allocEventId") + conflicts = " OR ".join( + f'(MIN("{column}") IS NOT MAX("{column}") OR ' + f'(COUNT("{column}")>0 AND COUNT("{column}")=?", [params["min_size"]] + if template == "freed_block_lifetime": + return "allocation_proved AND completion_proved AND freeEventId>=allocEventId", [] + clauses, values = [], [] + for field in ("id", "address", "size", "requestedSize", "allocEventId", "freeEventId"): + for prefix, operator in (("", "="), ("min_", ">="), ("max_", "<=")): + value = params.get(prefix + field) + if value is not None: + clauses.append(f'"{field}" {operator} ?') + values.append(value) + return " AND ".join(clauses) or "1", values + + +def lifecycle_query(sources, items, template, params, scope, candidate_limit): + required = _required_slices(sources, items, _attachment_limit()) + if required is None: + return None + predicate, values = _filter(template, params) + cte = _lifecycle_cte(sources, items, required) + with _AttachedSources(sources, required) as database: + summary = database.read( + cte + f"""SELECT COUNT(*) AS lifecycles, + COALESCE(SUM(allocation_proved),0) AS allocation_events_resolved, + COALESCE(SUM(completion_proved),0) AS free_events_resolved, + COALESCE(SUM(token LIKE 'unproved:%'),0) AS unproved_identities, + COALESCE(SUM(allocEventId>=0 AND NOT allocation_proved),0) AS missing_allocations, + COALESCE(MAX(invariant_conflict),0) AS invariant_conflict, + COALESCE(MAX(free_conflict),0) AS free_conflict, + COALESCE(SUM(CASE WHEN {predicate} THEN 1 ELSE 0 END),0) AS matching + FROM lifecycles""", + values, + )[0] + if summary["invariant_conflict"]: + raise QueryExecutionError("Conflicting observations for a proved lifecycle identity.") + if summary["free_conflict"]: + raise QueryExecutionError("Conflicting free-completion events for one lifecycle.") + scope["source_coverage"] = { + "coverage_scope": "all_deduplicated_lifecycles_before_filters_and_page", + "range_complete": True, + **{ + key: summary[key] + for key in ( + "lifecycles", + "allocation_events_resolved", + "free_events_resolved", + "unproved_identities", + ) + }, + "allocation_source_complete": not summary["missing_allocations"], + "source_queries": sources.query_count, + } + if template == "freed_block_lifetime": + buckets = database.read( + cte + f"""SELECT CASE + WHEN freeEventId-allocEventId<1000 THEN 0 + WHEN freeEventId-allocEventId<5000 THEN 1 + WHEN freeEventId-allocEventId<20000 THEN 2 + WHEN freeEventId-allocEventId<100000 THEN 3 ELSE 4 END AS bucket, + COUNT(*) AS block_count, SUM(size) AS size_bytes + FROM lifecycles WHERE {predicate} GROUP BY bucket ORDER BY bucket""", + values, + ) + for row in buckets: + row["lifetime_events"] = ("<1k", "1k-5k", "5k-20k", "20k-100k", ">=100k")[ + row.pop("bucket") + ] + scope["source_coverage"]["source_queries"] = sources.query_count + return buckets, len(buckets) + order = params["order_by"] if template == "block" else "size" + direction = params["order_dir"] if template == "block" else "DESC" + rows = database.read( + cte + f"SELECT * FROM lifecycles WHERE {predicate} " + f'ORDER BY "{order}" {direction}, id {direction}, token {direction} LIMIT ?', + [*values, candidate_limit], + ) + refs = sources.events( + cast(int, row[field]) + for row in rows + for field in ("allocEventId", "freeEventId") + if type(row[field]) is int and cast(int, row[field]) >= 0 + ) + for row in rows: + alloc, free = row["allocEventId"], row["freeEventId"] + allocation = refs.get(alloc) if row.pop("allocation_proved") else None + completion = refs.get(free) if row.pop("completion_proved") else None + token = row.pop("token") + row.pop("invariant_conflict") + row.pop("free_conflict") + row.update( + { + "lifecycle_id": f"{sources.namespace}:{token}", + "identity_status": ( + "proved_allocation_event" + if allocation + else ( + "stable_negative_token" + if token.startswith("preexisting:") + else "unproved_shard_local" + ) + ), + "allocation_source": allocation.to_dict() if allocation else None, + "allocation_source_status": ( + "resolved" + if allocation + else ("unknown_preexisting" if alloc == -1 else "missing") + ), + "free_source": completion.to_dict() if completion else None, + "free_source_status": ( + "resolved" + if completion + else ("live_or_unknown" if free is None or free < 0 else "missing") + ), + "state_scope": "latest_slice_observation", + "terminal_survivor": bool(row["terminal_survivor"]), + "category": "dynamic" if alloc >= 0 else "static" if free == -1 else "preexisting", + } + ) + scope["source_coverage"]["source_queries"] = sources.query_count + return rows, summary["matching"] diff --git a/tests/core/test_dataset_source_reuse.py b/tests/core/test_dataset_source_reuse.py index 07e6e32..e20baad 100644 --- a/tests/core/test_dataset_source_reuse.py +++ b/tests/core/test_dataset_source_reuse.py @@ -58,7 +58,7 @@ def test_sixty_thousand_events_are_not_read_twice(tmp_path, template): finally: service.close() assert result.returned == 1 - assert budget.work_rows == 60_001 + assert budget.work_rows <= 60_001 if template == "callstack_analysis": assert result.rows[0]["alloc_count"] == 60_000 @@ -198,8 +198,8 @@ def test_callstack_sort_loads_frames_only_for_returned_page(tmp_path): "event", params={"order_by": "callstack", "limit": 1}, db_path=root, _budget=budget ) assert result.rows[0]["frames"] == [{"name": "f"}] - # 600 trace rows + 7 schema rows + 1 coverage + 1 frame + 1 output. - assert budget.work_rows == 610 + # SQL pushdown may reduce trace work; only one returned frame is hydrated. + assert budget.work_rows <= 610 @pytest.mark.parametrize("invalid", [True, 1.0]) diff --git a/tests/core/test_dataset_sql_pushdown.py b/tests/core/test_dataset_sql_pushdown.py new file mode 100644 index 0000000..0e2672f --- /dev/null +++ b/tests/core/test_dataset_sql_pushdown.py @@ -0,0 +1,411 @@ +"""Bounded global SQL reductions preserve full-scope evidence and lifecycle proof.""" + +import dataclasses +import hashlib +import json +import sqlite3 +import time +from contextlib import closing + +import pytest + +from pt_snap_cli.core import dataset_lifecycle_sql +from pt_snap_cli.core.context_cache import ContextCache +from pt_snap_cli.core.dataset_global import global_query +from pt_snap_cli.core.dataset_resolver import DatasetResolver +from pt_snap_cli.core.dataset_sources import DatasetSourceResolver, QueryBudget +from pt_snap_cli.core.errors import QueryExecutionError, QueryTimeoutError +from pt_snap_cli.core.import_service import ImportService +from pt_snap_cli.core.query_service import QueryService +from pt_snap_cli.query.registry import get_query +from tests.core.test_dataset_attribution import case +from tests.core.test_dataset_import import source_options +from tests.test_dataset_focus import make_dataset + + +def large_dataset(tmp_path, per_shard=60_000): + root = make_dataset(tmp_path / "large", devices=(0,), slices=2) + manifest_path = root / "manifest.json" + manifest = json.loads(manifest_path.read_text()) + manifest["eventsPerSlice"] = per_shard + manifest["devices"]["0"]["eventCount"] = 2 * per_shard + for index, record in enumerate(manifest["devices"]["0"]["slices"]): + low, high = index * per_shard, (index + 1) * per_shard - 1 + record.update(startEventId=low, endEventId=high) + with closing(sqlite3.connect(root / record["file"])) as conn, conn: + conn.execute("DELETE FROM trace_entry_0") + conn.executemany( + "INSERT INTO trace_entry_0 VALUES (?,4,?,8,0,?,?,?,'shared')", + ((i, i * 8, i * 8, i * 8, i * 16) for i in range(low, high + 1)), + ) + # Carry-in live blocks are repeated; future allocations never appear. + conn.executemany( + "INSERT INTO block_0 VALUES (?,?,8,8,1,?,-1)", + ((i, i * 8, i) for i in range(high + 1)), + ) + manifest_path.write_text(json.dumps(manifest)) + return root + + +@pytest.fixture(scope="module") +def large_root(tmp_path_factory): + return large_dataset(tmp_path_factory.mktemp("sql-pushdown")) + + +@pytest.mark.parametrize( + "template", + ["memory_peak", "event", "allocation", "callstack_analysis", "block", "leak_detection"], +) +def test_large_real_scope_fetches_only_aggregates_and_bounded_candidates(large_root, template): + budget = QueryBudget(None, time.monotonic(), max_work_rows=64) + with closing(QueryService()) as service: + result = service.execute_query( + template, db_path=large_root, max_rows=5, exact_total=True, _budget=budget + ) + expected_total = 1 if template in ("memory_peak", "callstack_analysis") else 120_000 + assert result.total == expected_total + assert result.returned == min(expected_total, 5) + assert result.has_more == (expected_total > 5) + assert budget.work_rows <= 64 + if template in ("block", "leak_detection"): + coverage = result.scope["source_coverage"] + assert coverage["lifecycles"] == 120_000 + assert coverage["allocation_events_resolved"] == 120_000 + if template == "callstack_analysis": + assert result.rows[0]["alloc_count"] == 120_000 + assert result.rows[0]["total_size"] == 960_000 + + +def test_unbounded_output_still_exhausts_real_fetched_work_budget(large_root): + budget = QueryBudget(None, time.monotonic()) + with closing(QueryService()) as service: + with pytest.raises(QueryExecutionError, match="work budget"): + service.execute_query("allocation", db_path=large_root, _budget=budget) + assert budget.work_rows > budget.max_work_rows + + +def test_actual_range_precedes_filters_and_empty_window_is_exact(tmp_path): + root = make_dataset(tmp_path / "range", devices=(0,), slices=3) + with closing(QueryService()) as service: + result = service.execute_query( + "event", + {"min_id": 1, "max_id": 4, "action": 99, "limit": 0}, + db_path=root, + exact_total=True, + ) + assert result.rows == [] and result.total == 0 and not result.has_more + assert result.scope["actual_range"] == { + "first_event_id": 1, + "last_event_id": 4, + "real_event_count": 4, + } + + +def run_resolved(resolved, template, params=None, budget=None, slice_index=None): + config = get_query(template) + with closing(ContextCache()) as cache: + source = DatasetSourceResolver( + resolved, 0, cache, budget or QueryBudget(None, time.monotonic()) + ) + return global_query( + source, + template, + config.validate_params(params or {}), + 5, + True, + slice_index, + config.semantics_version, + ) + + +@pytest.mark.parametrize("template", ["block", "leak_detection", "freed_block_lifetime"]) +def test_attachment_ceiling_fallback_preserves_lifecycle_results(tmp_path, monkeypatch, template): + root = case(tmp_path, devices=(0,)) + resolved = DatasetResolver().inspect(root) + pushed = run_resolved(resolved, template) + monkeypatch.setattr(dataset_lifecycle_sql, "_attachment_limit", lambda: 1) + fallback = run_resolved(resolved, template) + assert pushed.rows == fallback.rows + assert pushed.total == fallback.total and pushed.has_more == fallback.has_more + first, second = pushed.scope["source_coverage"], fallback.scope["source_coverage"] + assert {k: v for k, v in first.items() if k != "source_queries"} == { + k: v for k, v in second.items() if k != "source_queries" + } + + +@pytest.mark.parametrize("template", ["block", "freed_block_lifetime"]) +@pytest.mark.parametrize("slice_index", [0, 3, 7]) +def test_slice_scope_pushdown_matches_fallback(tmp_path, monkeypatch, template, slice_index): + resolved = DatasetResolver().inspect(case(tmp_path, devices=(0,))) + pushed = run_resolved(resolved, template, slice_index=slice_index) + monkeypatch.setattr(dataset_lifecycle_sql, "_attachment_limit", lambda: 0) + fallback = run_resolved(resolved, template, slice_index=slice_index) + assert pushed.rows == fallback.rows + assert pushed.total == fallback.total + assert pushed.has_more == fallback.has_more + first, second = pushed.scope["source_coverage"], fallback.scope["source_coverage"] + assert {k: v for k, v in first.items() if k != "source_queries"} == { + k: v for k, v in second.items() if k != "source_queries" + } + assert pushed.scope["slice_indices"] == [slice_index] + + +@pytest.mark.parametrize("attachment_limit", [0, 10]) +def test_slice_leak_scope_is_rejected_before_pushdown_or_fallback( + tmp_path, monkeypatch, attachment_limit +): + root = make_dataset(tmp_path / "slice-leak", devices=(0,)) + monkeypatch.setattr(dataset_lifecycle_sql, "_attachment_limit", lambda: attachment_limit) + with closing(QueryService()) as service: + with pytest.raises(QueryExecutionError, match="requires terminal dataset scope"): + service.execute_query("leak_detection", db_path=root, slice_index=0) + + +def test_slice_attaches_only_observations_and_referenced_proof_owners(tmp_path, monkeypatch): + root = make_dataset(tmp_path / "many-slices", devices=(0,), slices=12) + with closing(sqlite3.connect(root / "device_0/slice_00011.db")) as conn, conn: + conn.executemany( + "INSERT INTO block_0 VALUES (?,?,?,?,?,?,?)", + [(0, 100, 8, 8, -1, 0, 2), (-1, 200, 8, 8, 1, -1, -1), (1, 300, 8, 8, 1, 1, -1)], + ) + with closing(sqlite3.connect(root / "device_0/slice_00000.db")) as conn, conn: + conn.execute("UPDATE trace_entry_0 SET action=5 WHERE id=1") + with closing(sqlite3.connect(root / "device_0/slice_00001.db")) as conn, conn: + conn.execute("UPDATE trace_entry_0 SET action=6 WHERE id=2") + resolved = DatasetResolver().inspect(root) + attached = [] + original = dataset_lifecycle_sql._AttachedSources.__enter__ + + def observed(database): + result = original(database) + attached.append( + [ + row[1] + for row in database.connection.execute("PRAGMA database_list") + if row[1] != "main" + ] + ) + return result + + monkeypatch.setattr(dataset_lifecycle_sql._AttachedSources, "__enter__", observed) + monkeypatch.setattr(dataset_lifecycle_sql, "_attachment_limit", lambda: 3) + pushed = run_resolved(resolved, "block", slice_index=11) + assert attached == [["s0", "s1", "s11"]] + row = next(row for row in pushed.rows if row["id"] == 0) + assert row["allocation_source"]["slice_index"] == 0 + assert row["free_source"]["slice_index"] == 1 + monkeypatch.setattr(dataset_lifecycle_sql, "_attachment_limit", lambda: 2) + fallback = run_resolved(resolved, "block", slice_index=11) + assert attached == [["s0", "s1", "s11"]] + assert pushed.rows == fallback.rows + assert pushed.total == fallback.total == 3 + + +def test_empty_slice_does_not_attach_unreferenced_members(tmp_path, monkeypatch): + root = make_dataset(tmp_path / "empty-slice", devices=(0,), slices=12) + resolved = DatasetResolver().inspect(root) + monkeypatch.setattr(dataset_lifecycle_sql, "_attachment_limit", lambda: 1) + with closing(ContextCache()) as cache: + sources = DatasetSourceResolver(resolved, 0, cache, QueryBudget(None, time.monotonic())) + assert dataset_lifecycle_sql._required_slices(sources, [sources.device.slices[11]], 1) == [ + sources.device.slices[11] + ] + assert run_resolved(resolved, "block", slice_index=11).rows == [] + + +@pytest.mark.parametrize("template", ["event", "block", "leak_detection"]) +@pytest.mark.parametrize("value", [-(2**63), 2**63 - 1]) +def test_int64_filter_endpoints_are_accepted(tmp_path, template, value): + root = case(tmp_path, devices=(0,)) + with closing(QueryService()) as service: + result = service.execute_query( + template, + {"min_allocated" if template == "event" else "min_size": value}, + db_path=root, + max_rows=1, + ) + assert result.returned == (1 if value < 0 else 0) + + +@pytest.mark.parametrize("template", ["event", "block", "leak_detection"]) +@pytest.mark.parametrize("value", [-(2**63) - 1, 2**63]) +def test_out_of_int64_filters_raise_domain_errors(tmp_path, template, value): + root = make_dataset(tmp_path / "overflow", devices=(0,)) + with closing(QueryService()) as service: + with pytest.raises(QueryExecutionError): + service.execute_query( + template, + {"min_allocated" if template == "event" else "min_size": value}, + db_path=root, + ) + + +@pytest.mark.parametrize("template", ["event", "block"]) +@pytest.mark.parametrize("offset", [2**63 - 2, 2**63 - 1, 2**63]) +def test_candidate_limit_int64_boundary(tmp_path, template, offset): + root = make_dataset(tmp_path / "offset", devices=(0,)) + with closing(QueryService()) as service: + if offset + 1 <= 2**63 - 1: + result = service.execute_query(template, {"offset": offset, "limit": 1}, db_path=root) + assert result.rows == [] + else: + with pytest.raises(QueryExecutionError): + service.execute_query(template, {"offset": offset, "limit": 1}, db_path=root) + + +def last_observation(root, block_id): + for path in sorted(root.glob("device_0/*.db"), reverse=True): + with closing(sqlite3.connect(path)) as conn: + if conn.execute("SELECT 1 FROM block_0 WHERE id=?", [block_id]).fetchone(): + return path + raise AssertionError(f"No observation for block {block_id}") + + +def test_lifecycle_conflicts_are_checked_before_filters_and_page(tmp_path): + root = case(tmp_path, devices=(0,)) + resolved = DatasetResolver().inspect(root) + with closing(sqlite3.connect(last_observation(root, 0))) as conn, conn: + assert conn.execute("UPDATE block_0 SET size=size+1 WHERE id=0").rowcount == 1 + with pytest.raises(QueryExecutionError, match="Conflicting observations"): + run_resolved(resolved, "block", {"id": 7}) + + +def test_latest_state_and_prior_proved_completion_survive_sql_reduction(tmp_path, monkeypatch): + root = case(tmp_path, devices=(0,)) + resolved = DatasetResolver().inspect(root) + with closing(sqlite3.connect(last_observation(root, 0))) as conn, conn: + assert conn.execute("UPDATE block_0 SET state=0,freeEventId=-1 WHERE id=0").rowcount == 1 + params = {"id": 0} + pushed = run_resolved(resolved, "block", params) + monkeypatch.setattr(dataset_lifecycle_sql, "_attachment_limit", lambda: 1) + fallback = run_resolved(resolved, "block", params) + assert pushed.rows == fallback.rows + assert pushed.rows[0]["state"] == 0 + assert pushed.rows[0]["freeEventId"] == 10 + assert pushed.rows[0]["free_source"]["event"]["action"] == 6 + assert not pushed.rows[0]["terminal_survivor"] + + +def test_attached_members_are_read_only_and_connection_is_closed(tmp_path): + resolved = DatasetResolver().inspect(make_dataset(tmp_path / "readonly", devices=(0,))) + with closing(ContextCache()) as cache: + source = DatasetSourceResolver(resolved, 0, cache, QueryBudget(None, time.monotonic())) + with dataset_lifecycle_sql._AttachedSources(source) as attached: + with pytest.raises(sqlite3.OperationalError, match="readonly"): + attached.connection.execute("DELETE FROM s0.trace_entry_0") + with pytest.raises(sqlite3.ProgrammingError, match="closed"): + attached.connection.execute("SELECT 1") + + +def test_attached_query_timeout_clears_handler_and_closes_connection(tmp_path): + resolved = DatasetResolver().inspect(make_dataset(tmp_path / "timeout", devices=(0,))) + with closing(ContextCache()) as cache: + source = DatasetSourceResolver(resolved, 0, cache, QueryBudget(0.05, time.monotonic())) + with dataset_lifecycle_sql._AttachedSources(source) as attached: + with pytest.raises(QueryTimeoutError): + attached.read( + "WITH RECURSIVE n(x) AS (SELECT 1 UNION ALL SELECT x+1 FROM n " + "WHERE x<100000000) SELECT SUM(x) FROM n" + ) + assert attached.connection.execute("SELECT 1").fetchone()[0] == 1 + with pytest.raises(sqlite3.ProgrammingError, match="closed"): + attached.connection.execute("SELECT 1") + + +def test_callstack_order_reuses_already_fetched_candidate_events(tmp_path, monkeypatch): + root = make_dataset(tmp_path / "reuse", devices=(0,), slices=3) + queries = [] + original = DatasetSourceResolver.read + + def observed(self, path, sql, values=None): + queries.append(sql) + return original(self, path, sql, values) + + monkeypatch.setattr(DatasetSourceResolver, "read", observed) + with closing(QueryService()) as service: + result = service.execute_query("event", {"order_by": "callstack", "limit": 1}, db_path=root) + assert result.returned == 1 + assert sum("ORDER BY" in sql and "trace_entry" in sql for sql in queries) == 3 + assert not any("WHERE t.id IN" in sql for sql in queries) + + +def test_attaching_locked_member_respects_remaining_deadline(tmp_path, monkeypatch): + root = make_dataset(tmp_path / "locked", devices=(0,), slices=1) + resolved = DatasetResolver().inspect(root) + # Exercise the native non-immutable opening mode with an actual SQLite lock. + monkeypatch.setattr(type(resolved), "callstack_layout", property(lambda self: "v2")) + with closing(sqlite3.connect(root / "device_0/slice_00000.db")) as locked: + locked.execute("BEGIN EXCLUSIVE") + with closing(ContextCache()) as cache: + start = time.monotonic() + source = DatasetSourceResolver(resolved, 0, cache, QueryBudget(0.05, start)) + attached = dataset_lifecycle_sql._AttachedSources(source) + with pytest.raises(QueryTimeoutError): + with attached: + attached.read("SELECT COUNT(*) FROM s0.trace_entry_0") + assert time.monotonic() - start < 0.5 + with pytest.raises(sqlite3.ProgrammingError, match="closed"): + attached.connection.execute("SELECT 1") + locked.rollback() + + +@pytest.mark.parametrize("layout", ["v1", "v2"]) +@pytest.mark.parametrize("collation,texts", [("NOCASE", ("a", "A")), ("RTRIM", ("a ", "a"))]) +def test_global_text_identity_and_candidates_ignore_source_collation( + tmp_path, layout, collation, texts +): + if layout == "v1": + root = make_dataset(tmp_path / "collated", devices=(0,), slices=1) + else: + root = ( + ImportService() + .import_snapshot( + dataclasses.replace(source_options(tmp_path), events_per_slice=99, set_focus=False) + ) + .db_path + ) + manifest_path = root / "manifest.json" + manifest = json.loads(manifest_path.read_text()) + item = manifest["devices"]["0"]["slices"][0] + member = root / item["file"] + with closing(sqlite3.connect(member)) as conn, conn: + ids = [r[0] for r in conn.execute("SELECT id FROM trace_entry_0 WHERE id>=0 ORDER BY id")] + if layout == "v1": + original = conn.execute( + "SELECT sql FROM sqlite_master WHERE name='trace_entry_0'" + ).fetchone()[0] + conn.execute("ALTER TABLE trace_entry_0 RENAME TO original_trace") + conn.execute(original.replace("callstack TEXT", f"callstack TEXT COLLATE {collation}")) + conn.execute("INSERT INTO trace_entry_0 SELECT * FROM original_trace") + conn.execute("DROP TABLE original_trace") + conn.execute("UPDATE trace_entry_0 SET callstack=?", [texts[0]]) + conn.execute("UPDATE trace_entry_0 SET callstack=? WHERE id=?", [texts[1], ids[1]]) + else: + conn.execute("DROP TABLE callstack") + conn.execute( + f"CREATE TABLE callstack (id INTEGER PRIMARY KEY,callstack TEXT COLLATE {collation})" + ) + conn.executemany("INSERT INTO callstack VALUES (?,?)", enumerate(texts)) + conn.execute("UPDATE trace_entry_0 SET callstackId=0") + conn.execute("UPDATE trace_entry_0 SET callstackId=1 WHERE id=?", [ids[1]]) + conn.execute("UPDATE trace_entry_0 SET size=1 WHERE id>=0") + if layout == "v2": + item["sha256"] = hashlib.sha256(member.read_bytes()).hexdigest() + manifest_path.write_text(json.dumps(manifest)) + expected_text = {event: texts[int(event == ids[1])] for event in ids} + with closing(QueryService()) as service: + groups = service.execute_query("callstack_analysis", {"min_count": 1}, db_path=root) + assert {r["callstack"]: r["alloc_count"] for r in groups.rows} == { + texts[0]: len(ids) - 1, + texts[1]: 1, + } + assert len({r["source_stack_id"] for r in groups.rows}) == 2 + for direction in ("ASC", "DESC"): + result = service.execute_query( + "event", {"order_by": "callstack", "order_dir": direction, "limit": 1}, db_path=root + ) + expected = sorted( + ids, key=lambda event: (expected_text[event], event), reverse=direction == "DESC" + )[0] + assert result.rows[0]["id"] == expected + assert result.rows[0]["callstack"] == expected_text[expected] diff --git a/tests/test_contract_cli_api.py b/tests/test_contract_cli_api.py index 0cf9cfd..ef0c9d1 100644 --- a/tests/test_contract_cli_api.py +++ b/tests/test_contract_cli_api.py @@ -22,7 +22,7 @@ from pt_snap_cli.api import FocusState, SnapshotAnalyzer from pt_snap_cli.cli import app -from pt_snap_cli.core import InvalidDeviceError, TemplateRenderError +from pt_snap_cli.core import InvalidDeviceError, QueryExecutionError, TemplateRenderError from pt_snap_cli.core.json_codec import JSON_SCHEMA_VERSION from pt_snap_cli.query.config import QueryParameter, QueryTemplate from pt_snap_cli.query.registry import QueryRegistry, register_query @@ -33,6 +33,25 @@ pytestmark = pytest.mark.usefixtures("owned_service_instances") +@pytest.mark.parametrize("template", ["event", "block"]) +@pytest.mark.parametrize("params", [{"min_size": 2**63}, {"offset": 2**63, "limit": 1}]) +def test_dataset_int64_overflow_cli_api_domain_error(tmp_path, template, params): + if template == "event" and "min_size" in params: + params = {"min_allocated": params["min_size"]} + root = make_dataset(tmp_path / "int64-overflow", devices=(0,)) + with SnapshotAnalyzer(root) as analyzer: + with pytest.raises(QueryExecutionError): + analyzer.execute_query(template, params) + result = runner.invoke( + app, + ["query", str(root), "--template-use", template, "--params", json.dumps(params), "--json"], + ) + assert result.exit_code == 1 + assert result.stdout == "" + assert json.loads(result.stderr)["error"]["code"] == "QUERY_FAILED" + assert "Traceback" not in result.stderr + + @pytest.mark.parametrize("template", ["event", "active_memory_callstack_at_event"]) def test_stack_summary_cli_api_and_catalog_contract(contract_db: Path, template: str): with closing(sqlite3.connect(contract_db)) as conn, conn: