Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,9 @@ canonical stack statistics, deduplicated lifecycles, terminal leak candidates an
`report peak-memory` share bounded core semantics; custom SQL/overrides remain unsupported
on datasets. See the [global support matrix](docs/en/querying.md#dataset-global-built-in-support-p3)
and [dataset attribution and coverage](docs/en/querying.md#dataset-point-event-attribution).
Lifecycle queries exceeding SQLite's attachment limit use a private, page-limited
temporary derived database; source shards stay read-only. See the global support
matrix for storage limits, cleanup and memory caveats.

Explicit `pt-snap import snapshot.pkl --format msinsight --json` produces
`snapshot.pkl.msinsight/` for the fixed msinsight revision's **same original
Expand Down
2 changes: 2 additions & 0 deletions README_zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,8 @@ pt-snap split snapshot.pkl --slices 4 --output snapshot-slices
终片泄漏候选及 `report peak-memory` 共用有界 core 语义;数据集仍不支持自定义 SQL/override。
详见[全局支持矩阵](docs/zh/querying.md#数据集全局内建支持p3)及
[数据集定点归因与覆盖](docs/zh/querying.md#数据集定点事件归因)。
超过 SQLite 附加上限的生命周期查询使用私有、限制主库页数空间的临时派生数据库,源分片
保持只读;存储限额、清理及内存边界见全局支持矩阵。

显式 `pt-snap import snapshot.pkl --format msinsight --json` 生成
`snapshot.pkl.msinsight/`,对齐固定 msinsight 版本的**同一原始 pickle + 邻接缓存**入口。
Expand Down
31 changes: 24 additions & 7 deletions docs/en/querying.md
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,8 @@ check cancellation. Individual filesystem calls are not OS-level hard-preempted.
`PT_SNAP_DATASET_CACHE=immutable` explicitly opts into metadata-only validation reuse
only when the caller guarantees immutable finalized files and reliable filesystem
change times; see [dataset validation and cache contracts](sharded-snapshotdb.md).
No temporary merge DB, repair or artifact write.
Source artifacts are never repaired or written. A large lifecycle query may create a
private temporary derived database as described below.

The compatible exporter defaults to **500000 real events per device shard**.
That storage capacity is separate from the **100000-row / 64-MiB query work
Expand All @@ -180,16 +181,32 @@ formatted text is not their identity. Block/leak/lifetime reductions use read-on
attached shards for proof, conflicts and latest observations before pagination;
the attachment limit (normally ten) counts selected observation shards plus any
other shards owning their allocation/free references. Slice queries discover those
proof owners before filtering or paging; exceeding the limit retains the budgeted
source-reading path. SQLite internal scans and temporary space are not bounded by
the fetched-row/byte budget. Summary and page (or lifetime buckets) each evaluate
the full lifecycle CTE, sharing the same deadline. No intermediate merged database
is created.
proof owners before filtering or paging. Above the limit, a query-private temporary
SQLite database retains only distinct event references and one narrow state row per
lifecycle. Sources are attached read-only one at a time: reference ownership/actions
are proved first, then all observations are reduced and conflicts checked before
filters, totals or pagination. Only returned candidates hydrate complete source events.
The small attached-set path is unchanged.

The temporary directory is private (0700), its derived database is created with 0600
permissions, and normal completion, errors and cancellation close connections and
remove the directory. A killed process or failed filesystem cleanup can leave private
residue; this is not secure erasure. Its main database page ceiling defaults to
512 MiB, configurable through the internal `QueryBudget.max_scratch_db_bytes` field from
4096 bytes to 4 GiB; zero disables this path and retains the original budgeted source
reader. This is not a new CLI option. Exceeding the page ceiling or running out of
storage rejects the entire operation. The ceiling does **not** bound rollback journals,
SQLite sorting files, total temporary space or RSS. Main/source page caches request
2 MiB each; these are cache targets, not memory guarantees. Internal SQL scans and
temporary work are not counted as Python-fetched rows. All stages, summary/page or
lifetime buckets and source hydration share the same diminishing deadline.

A million-event dataset can therefore answer aggregates and small pages within the
default work budget. Use `-n 5` or an explicit finite `limit` for large row-returning
queries. Unlimited output, large offsets, many distinct stack groups, ordered-frame
reads or the attachment-limit fallback can still exceed the budget and fail honestly.
reads or a disabled temporary reducer can still exceed the fetched budget and fail
honestly. Many-lifecycle reductions can independently exceed the derived-page limit
or deadline.
The limit counts rows fetched into Python and serialized output, not internal SQL
scans, SQLite sorting space or total process memory.

Expand Down
23 changes: 17 additions & 6 deletions docs/zh/querying.md
Original file line number Diff line number Diff line change
Expand Up @@ -136,7 +136,8 @@ SQLite、一个批次及单个 cell 有额外开销。默认仍执行强准入
分片数),新 CLI 进程仍冷启动。准入共用该 deadline:hash 分块、Python 行批次及 SQLite progress handler 都检查取消。
单个文件系统调用不提供 OS 级硬抢占。仅当调用方保证最终产物不可变且文件系统变更时间
可靠时,才可通过 `PT_SNAP_DATASET_CACHE=immutable` 显式启用纯元数据校验复用;参见
[数据集校验与缓存合同](sharded-snapshotdb.md)。不生成合并临时 DB、不修复或写产物。
[数据集校验与缓存合同](sharded-snapshotdb.md)。不修复或写入源产物;较大的生命周期查询
可能生成下述私有临时派生数据库。

兼容导出器默认**每设备每片 500000 个真实事件**。该存储容量与 **100000 行 / 64 MiB
查询工作预算**独立:即使只有一个默认大小的分片,无足够筛选或分页的结果物化也可能
Expand All @@ -149,13 +150,23 @@ SQLite、一个批次及单个 cell 有额外开销。默认仍执行强准入
与总计数计算。规范化原始有序帧的身份不能用格式化文本替代,因此仍走受预算约束的来源
读取路径。block/leak/lifetime 通过只读附加分片先证明身份、检测冲突、选最新观察,再分页;
附加上限(通常十片)按选定观察分片与其分配/释放引用所属的其他分片合计。
slice 查询在筛选、分页前发现所需的证明来源分片;超限时回退到原有受预算约束的来源
读取路径。SQLite 内部扫描与临时空间不受取回行数/字节预算约束;汇总和分页(或存活时长
分桶)各自执行完整的生命周期 CTE,共用同一 deadline。不创建中间合并数据库。
slice 查询在筛选、分页前发现所需的证明来源分片。超限时使用查询专属的临时 SQLite
派生库,仅保留去重后的事件引用及每个生命周期的一条窄状态记录。源分片逐个只读挂载:
先证明引用的所属片及 action,再归约全部观察、检查冲突,之后才筛选、计数和分页。
仅返回候选补全完整来源事件。未超过附加上限的原路径不变。

临时目录权限为 0700,派生文件以 0600 创建;正常完成、错误和取消均关闭连接并删除目录。
强制杀进程或文件系统清理失败可能留下私有残余;这不是安全擦除。派生主库页数空间默认
上限为 512 MiB,可通过内部 `QueryBudget.max_scratch_db_bytes` 字段设为 4096 字节至
4 GiB;0 禁用该路径、沿用原有受预算约束的来源读取。这不是新增 CLI 选项。页数超限或
磁盘满会拒绝整个查询。该上限**不包括**回滚 journal、SQLite 排序文件、全部临时空间或
RSS。主库和源库各请求 2 MiB page cache,仅为缓存目标,不保证内存上限。SQL 内部扫描
和临时计算不算 Python 取回行;全部阶段、汇总/分页或时长分桶及来源补全共用递减 deadline。

因此,百万事件数据集的聚合与小页面可在默认工作预算内完成。大量逐行结果请指定 `-n 5`
或有限 `limit`。无限制输出、大 offset、大量不同栈分组、有序帧读取及超过附加上限的回退
路径仍可能超预算,并明确失败。预算统计进入 Python 的实际取回行和序列化输出,不限制
或有限 `limit`。无限制输出、大 offset、大量不同栈分组、有序帧读取及禁用临时归约库后的
来源读取仍可能超取回预算,并明确失败。大量生命周期归约也可能单独超出派生主库页数上限
或 deadline。预算统计进入 Python 的实际取回行和序列化输出,不限制
SQL 内部扫描、SQLite 排序空间或整个进程内存。

## 数据集定点事件归因
Expand Down
2 changes: 1 addition & 1 deletion src/pt_snap_cli/core/dataset_global.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
"""Exact built-in global reductions, without a temporary merge database.
"""Exact built-in global reductions, with bounded fetched work.

Shard SQL reduces counters, groups, and candidate windows before Python merging.
Fetched work is cumulatively bounded by QueryBudget, including source hydration
Expand Down
88 changes: 88 additions & 0 deletions src/pt_snap_cli/core/dataset_lifecycle_connection.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
"""Shared deadline-aware SQLite connection ownership for lifecycle reductions."""

from __future__ import annotations

import math
import sqlite3
import time

from pt_snap_cli.core.dataset_contract import SliceRecord
from pt_snap_cli.core.dataset_files import require_readonly_member
from pt_snap_cli.core.dataset_sources import DatasetSourceResolver
from pt_snap_cli.core.errors import QueryExecutionError


class _AttachedSources:
def __init__(
self,
sources: DatasetSourceResolver,
items: list[SliceRecord] | None = None,
*,
database: str = ":memory:",
) -> None:
self.sources = sources
self.items = sources.device.slices if items is None else items
remaining = sources.budget.remaining()
self.connection = sqlite3.connect(
database, uri=True, timeout=min(5.0, remaining) if remaining is not None else 5.0
)
self.connection.row_factory = sqlite3.Row

def _configure_deadline(self):
budget = self.sources.budget
remaining = budget.remaining()
deadline = budget.started + budget.timeout_s if budget.timeout_s is not None else None
self.connection.set_progress_handler(
(lambda: int(time.monotonic() >= deadline)) if deadline is not None else None, 1000
)
# SQLite's default five-second lock wait must not outlive the operation.
wait_ms = math.ceil(1000 * min(5.0, remaining)) if remaining is not None else 5000
self.connection.execute(f"PRAGMA busy_timeout={wait_ms}")

def __enter__(self):
try:
self._configure_deadline()
self.connection.execute("PRAGMA trusted_schema=OFF")
for item in self.items:
self._configure_deadline()
path = self.sources.dataset.root / item.file
immutable = self.sources.dataset.callstack_layout == "v1"
require_readonly_member(self.sources.dataset.root, path, immutable=immutable)
uri = path.as_uri() + "?mode=ro" + ("&immutable=1" if immutable else "")
self.connection.execute(f'ATTACH DATABASE ? AS "s{item.index}"', [uri])
self.connection.execute("PRAGMA query_only=ON")
self.sources.budget.remaining()
return self
except BaseException as exc:
self.connection.set_progress_handler(None, 0)
self.connection.close()
if isinstance(exc, sqlite3.Error):
self.sources.budget.remaining()
raise QueryExecutionError(str(exc)) from exc
raise

def __exit__(self, exc_type, exc_value, traceback):
self.connection.set_progress_handler(None, 0)
self.connection.close()
return False

def read(self, sql: str, values: list[object] | None = None):
budget = self.sources.budget
cursor = self.connection.cursor()
self.sources.query_count += 1
try:
self._configure_deadline()
cursor.execute(sql, values or [])
result = []
while batch := cursor.fetchmany(256):
rows = [dict(row) for row in batch]
budget.consume(rows)
result.extend(rows)
budget.remaining()
return result
except (sqlite3.Error, OverflowError) as exc:
budget.remaining() # Preserve the domain timeout on interruption.
raise QueryExecutionError(str(exc)) from exc
finally:
cursor.close()
self.connection.set_progress_handler(None, 0)
Loading
Loading