Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,12 @@ text-only databases require re-importing the original snapshot.
`report memory-tree` also accepts `--json`. The HTML view has full-tree search and
keyboard-accessible child lists, so tiny and zero-byte frames remain reachable.

Dataset work limits are configurable with `--max-work-rows`, `--max-work-bytes`,
`--max-scratch-db-bytes`, environment variables, or immutable Python `QueryLimits`.
Defaults remain 100000 work rows / 64 MiB work bytes / 512 MiB scratch pages; see
[public dataset limits](docs/en/querying.md#public-dataset-limits) for precedence,
deep-pagination costs and all-or-error semantics.

## Agent Skills

For long diagnostic stacks, opt into `stack_bytes` on `event` or
Expand Down
5 changes: 5 additions & 0 deletions README_zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,11 @@ included bytes 为分母。解读有上限的分组结果前,请参阅
`report memory-tree` 同样支持 `--json`。HTML 支持全树搜索和可键盘操作的子节点列表,
极小占用及零字节帧也能进入查看。

数据集工作预算可通过 `--max-work-rows`、`--max-work-bytes`、
`--max-scratch-db-bytes`、环境变量或不可变 Python `QueryLimits` 配置。
默认仍为 100000 工作行 / 64 MiB 工作字节 / 512 MiB scratch 页;优先级、
深分页成本和整次失败语义见[公开数据集预算](docs/zh/querying.md#公开数据集预算)。

## Agent Skills

诊断长调用栈时,可在 `event` 或 `active_memory_callstack_at_event` 上显式设置
Expand Down
55 changes: 52 additions & 3 deletions docs/en/querying.md
Original file line number Diff line number Diff line change
Expand Up @@ -149,7 +149,7 @@ marked, never guessed by address/local stack ID and never asserted as proved lea
Dataset reads detach batches before foreign-owner lookups, using the borrowed bounded
Context LRU. One diminishing deadline includes all shards, sources, grouping and exact
totals; reports explicitly share **one** budget across peak selection and attribution. Cumulative
fetched/output work is limited to **100000 rows / 64 MiB serialized values**, including
fetched/output work defaults to **100000 rows / 64 MiB serialized values**, including
repeated source reads. Exceeding either fails the entire operation, never an alleged
complete partial result. Python materialization/sort are bounded by this work ceiling, **not**
by max_rows. This is not a process RSS limit: Python/SQLite, one batch and individual
Expand Down Expand Up @@ -200,9 +200,10 @@ The temporary directory is private (0700), its derived database is created with
permissions, and normal completion, errors and cancellation close connections and
remove the directory. A killed process or failed filesystem cleanup can leave private
residue; this is not secure erasure. Its main database page ceiling defaults to
512 MiB, configurable through the internal `QueryBudget.max_scratch_db_bytes` field from
512 MiB, configurable with `--max-scratch-db-bytes`,
`PT_SNAP_MAX_SCRATCH_DB_BYTES` or public `QueryLimits.max_scratch_db_bytes` from
4096 bytes to 4 GiB; zero disables this path and retains the original budgeted source
reader. This is not a new CLI option. Exceeding the page ceiling or running out of
reader. See [public dataset limits](#public-dataset-limits). Exceeding the page ceiling or running out of
storage rejects the entire operation. The ceiling does **not** bound rollback journals,
SQLite sorting files, total temporary space or RSS. Main/source page caches request
2 MiB each; these are cache targets, not memory guarantees. Internal SQL scans and
Expand All @@ -218,6 +219,54 @@ or deadline.
The limit counts rows fetched into Python and serialized output, not internal SQL
scans, SQLite sorting space or total process memory.

## Public dataset limits

`query`, `report peak-memory` and `report memory-tree` accept the same options:

| CLI option | Environment variable | Default / valid values |
| --- | --- | --- |
| `--max-work-rows` | `PT_SNAP_MAX_WORK_ROWS` | `100000`; positive integer |
| `--max-work-bytes` | `PT_SNAP_MAX_WORK_BYTES` | `67108864` (64 MiB); positive integer bytes |
| `--max-scratch-db-bytes` | `PT_SNAP_MAX_SCRATCH_DB_BYTES` | `536870912` (512 MiB); `0` disables scratch, otherwise `4096` through `4294967296` (4 GiB) |

Each explicit flag overrides its environment variable, then the built-in default.
Environment values are decimal integers without units; unset or blank uses the default.
Booleans, fractions, negative values and zero work limits are errors; there is no
unlimited work-budget setting. Scratch zero **does not disable the work budget**.
Non-page-aligned scratch limits are rounded down to whole 4096-byte pages by SQLite.
Invalid configuration fails before database access, including on standalone commands.
These limits govern dataset execution; they do not add a new work cap to standalone
SQLite queries or make a dataset-unsupported template supported.

```bash
pt-snap query '<dataset_dir>' --template-use event \
--params '{"offset":100000,"limit":5}' --max-work-rows 300000 \
--max-work-bytes 134217728 --exact-total --json
pt-snap report peak-memory '<dataset_dir>' --max-work-rows 300000 \
--max-scratch-db-bytes 1073741824 --timeout 30 --json
```

Use the [immutable Python `QueryLimits` API](snapshot-analyzer-api.md#dataset-query-limits)
for the same settings in `SnapshotAnalyzer`, `QueryService` and `ReportService`.
A successful dataset query/report returns actual effective limits in
`scope.work_budget`: `max_rows`, `max_bytes`, `max_scratch_db_bytes`, and units.
Template `dataset_support` reports effective `work_limit_rows`, `work_limit_bytes`
and `max_scratch_db_bytes`, including environment or analyzer defaults; query
`--template-info` also honors explicit limit flags. Limits in an operation's scope
are authoritative for that operation, even if the environment later changes.
Budget and scratch exhaustion errors identify the configured limit and reject the
whole operation without a partial global result. Work exhaustion also reports
consumed rows/bytes. Peak-report steps share one cumulative budget and deadline;
each independent query/report starts fresh counters, even with reused connections.

Raising limits explicitly permits more work and may increase memory/disk costs; it
is not a performance optimization or an RSS/total-temporary-space guarantee.
On event/allocation pagination paths that fetch `offset + limit` candidates
per shard before global sorting and slicing, that full candidate work is charged. A small output can therefore exceed
the work ceiling; `-n`, `limit`, `stack_bytes` and `timeout` do not raise it. Prefer
narrower ranges when suitable, or deliberately size the limits for the operation.
Raising a budget does not optimize its pagination plan or silently change default limits.

## Dataset point-event attribution

A complete compatibility-v1 or native-v2 directory/manifest supports `event`
Expand Down
42 changes: 42 additions & 0 deletions docs/en/snapshot-analyzer-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,48 @@ and `device_id` is its default explicit query-device override. When `db_path` is
omitted, the database uses the same resolution order as the CLI:
`PT_SNAP_DB_PATH`, the nearest `.pt-snap/focus.json`, then legacy global config.

## Dataset query limits

Import the frozen `QueryLimits` configuration alongside `SnapshotAnalyzer` (or
from `pt_snap_cli.core` for direct services). It contains only three immutable
limits, never shared mutable counters:

```python
from pt_snap_cli.api import QueryLimits, SnapshotAnalyzer

limits = QueryLimits.resolve(max_work_rows=300_000, max_work_bytes=128 * 1024**2)
with SnapshotAnalyzer(Path("/path/to/dataset"), query_limits=limits) as analyzer:
result = analyzer.execute_query("event", {"offset":100_000, "limit":5})
print(result["scope"]["work_budget"])
# One-call override; subsequent calls still use the constructor's limits.
other = analyzer.execute_query("event", query_limits=QueryLimits(max_work_rows=400_000))
```

`QueryLimits()` always uses built-in defaults: `max_work_rows=100000`,
`max_work_bytes=67108864`, `max_scratch_db_bytes=536870912`.
`QueryLimits.resolve(...)` resolves each omitted field from its corresponding
`PT_SNAP_MAX_WORK_ROWS`, `PT_SNAP_MAX_WORK_BYTES`, or `PT_SNAP_MAX_SCRATCH_DB_BYTES`
environment value, then the default. Explicit fields ignore even invalid matching
environment values. Work limits must be positive integers (not booleans/floats);
scratch accepts integer zero to disable or 4096 through 4294967296 bytes. No
unlimited work setting exists. Invalid values raise core `InvalidParameterError`.

For each operation, a `query_limits` argument replaces the entire constructor
configuration; constructor configuration replaces environment/default resolution.
When neither is supplied, the environment is read afresh for each operation.
Use `QueryLimits.resolve(...)` if a partial override should retain other environment
settings. `get_template_info(..., query_limits=...)` can inspect a one-call override;
`list_capabilities()` reflects the analyzer's normal effective defaults.

`QueryService` and `ReportService` likewise accept constructor `query_limits`.
`execute_query`, `peak_memory_report`, `event_attribution`, and `memory_tree_report`
accept per-call `query_limits`. Peak-report steps share one cumulative counter and
deadline; independent operations start fresh counters. Passing mutable internal
`QueryBudget` objects is not the public configuration API. These limits only bound
dataset work, not standalone query materialization, RSS, or all SQLite temporary
space. See [configuration and failure semantics](querying.md#public-dataset-limits),
including deep-pagination costs and the unchanged default caps.

## Connection Lifetime and Cache Ownership

An analyzer reuses read-only SQLite connections for queries and overviews. Use
Expand Down
48 changes: 45 additions & 3 deletions docs/zh/querying.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,7 @@ exactness 标记诚实。`has_more/truncated` 描述窗口,不表示来源/范
独立。未证明历史身份按片局部单独表达,不按地址/局部栈 ID 猜合并,不断言为已证明泄漏。

数据集读取先分离批次再查外片来源,使用借用的有界 Context LRU。所有片、来源、分组、
精确总数共用递减 deadline;报告的峰值选择与归因明确共用**一个**预算。累计取回/输出工作上限为
精确总数共用递减 deadline;报告的峰值选择与归因明确共用**一个**预算。累计取回/输出工作默认上限为
**100000 行 / 64 MiB 序列化值**,含重复来源读取;超限整体失败,不返回冒充完整的部分
结论。物化/排序由工作上限约束,**不由 max_rows 约束**;不是进程 RSS 上限,Python/
SQLite、一个批次及单个 cell 有额外开销。默认仍执行强准入:每次对 manifest/全部成员
Expand Down Expand Up @@ -162,8 +162,9 @@ block 观察超过 4096 条时,使用查询专属的临时 SQLite

临时目录权限为 0700,派生文件以 0600 创建;正常完成、错误和取消均关闭连接并删除目录。
强制杀进程或文件系统清理失败可能留下私有残余;这不是安全擦除。派生主库页数空间默认
上限为 512 MiB,可通过内部 `QueryBudget.max_scratch_db_bytes` 字段设为 4096 字节至
4 GiB;0 禁用该路径、沿用原有受预算约束的来源读取。这不是新增 CLI 选项。页数超限或
上限为 512 MiB,可通过 `--max-scratch-db-bytes`、`PT_SNAP_MAX_SCRATCH_DB_BYTES`
或公开 `QueryLimits.max_scratch_db_bytes` 设为 4096 字节至 4 GiB;0 禁用该路径、
沿用原有受预算约束的来源读取。详见[公开数据集预算](#公开数据集预算)。页数超限或
磁盘满会拒绝整个查询。该上限**不包括**回滚 journal、SQLite 排序文件、全部临时空间或
RSS。主库和源库各请求 2 MiB page cache,仅为缓存目标,不保证内存上限。SQL 内部扫描
和临时计算不算 Python 取回行;全部阶段、汇总/分页或时长分桶及来源补全共用递减 deadline。
Expand All @@ -174,6 +175,47 @@ RSS。主库和源库各请求 2 MiB page cache,仅为缓存目标,不保证
或 deadline。预算统计进入 Python 的实际取回行和序列化输出,不限制
SQL 内部扫描、SQLite 排序空间或整个进程内存。

## 公开数据集预算

`query`、`report peak-memory` 与 `report memory-tree` 接受同一组选项:

| CLI 选项 | 环境变量 | 默认值 / 合法范围 |
| --- | --- | --- |
| `--max-work-rows` | `PT_SNAP_MAX_WORK_ROWS` | `100000`;正整数 |
| `--max-work-bytes` | `PT_SNAP_MAX_WORK_BYTES` | `67108864`(64 MiB);正整数字节数 |
| `--max-scratch-db-bytes` | `PT_SNAP_MAX_SCRATCH_DB_BYTES` | `536870912`(512 MiB);`0` 禁用 scratch,否则为 `4096` 至 `4294967296`(4 GiB) |

逐字段按显式 CLI flag、对应环境变量、内建默认值依次解析。环境变量只接受十进制整数,
不接受单位后缀;未设置或空白时使用默认值。布尔值、小数、负数和零工作预算都非法;
不存在无限工作预算。scratch 为零**不会禁用工作预算**。非整页 scratch 值由 SQLite
向下取整到 4096 字节页。非法配置在访问数据库前失败,单库命令也执行配置校验。
这些预算约束数据集执行,不给单库 SQLite 查询新增工作上限,也不让不支持数据集的模板
变得可用。

```bash
pt-snap query '<dataset_dir>' --template-use event \
--params '{"offset":100000,"limit":5}' --max-work-rows 300000 \
--max-work-bytes 134217728 --exact-total --json
pt-snap report peak-memory '<dataset_dir>' --max-work-rows 300000 \
--max-scratch-db-bytes 1073741824 --timeout 30 --json
```

`SnapshotAnalyzer`、`QueryService` 和 `ReportService` 共用
[不可变 Python `QueryLimits` API](snapshot-analyzer-api.md#数据集查询预算)。成功的数据集
查询/报告在 `scope.work_budget` 回显实际 `max_rows`、`max_bytes`、
`max_scratch_db_bytes` 及单位。模板 `dataset_support` 中的 `work_limit_rows`、
`work_limit_bytes` 与 `max_scratch_db_bytes` 反映当前环境或 analyzer 默认配置;
`query --template-info` 也接受显式预算 flag。单次结果的 scope 是该次有效预算的依据,
不受后续环境变更影响。工作/scratch 超限报错包含实际配置值,整次失败且不返回部分全局
结论;工作超限还报告已消耗的行数/字节数。峰值报告全部步骤共用累计预算与 deadline;
独立查询/报告每次创建新计数器,即使底层复用连接。

显式提高预算允许更多工作,可能增加内存/磁盘开销;这不是性能优化,也不保证 RSS 或
全部临时空间上限。对先从每片取 `offset + limit` 个候选、全局排序后再切页的
event/allocation 分页路径,全部候选工作仍计费,因此输出少也可能超预算。`-n`、`limit`、`stack_bytes`、`timeout`
不会提高工作预算。适用时优先缩小范围,或明确为该操作选择更大容量。提高预算本身
不会优化分页计划,也不暗中放宽默认预算。

## 数据集定点事件归因

完整兼容 v1 或原生 v2 目录/manifest 支持 `event` 寻址,以及接受真实 `event_id` 的
Expand Down
36 changes: 36 additions & 0 deletions docs/zh/snapshot-analyzer-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,42 @@ analyzer = SnapshotAnalyzer(
显式设备覆盖。省略 `db_path` 时,数据库解析顺序与 CLI 相同:`PT_SNAP_DB_PATH`、
最近的 `.pt-snap/focus.json`,最后是 legacy 全局配置。

## 数据集查询预算

可从 `pt_snap_cli.api` 与 `SnapshotAnalyzer` 一起导入冻结的 `QueryLimits`,直接调用
服务时也可从 `pt_snap_cli.core` 导入。它只包含三个不可变上限,不含共享可变计数器:

```python
from pt_snap_cli.api import QueryLimits, SnapshotAnalyzer

limits = QueryLimits.resolve(max_work_rows=300_000, max_work_bytes=128 * 1024**2)
with SnapshotAnalyzer(Path("/path/to/dataset"), query_limits=limits) as analyzer:
result = analyzer.execute_query("event", {"offset":100_000, "limit":5})
print(result["scope"]["work_budget"])
# 只覆盖本次调用;后续仍使用构造时的配置。
other = analyzer.execute_query("event", query_limits=QueryLimits(max_work_rows=400_000))
```

`QueryLimits()` 始终使用内建默认值:`max_work_rows=100000`、
`max_work_bytes=67108864`、`max_scratch_db_bytes=536870912`。
`QueryLimits.resolve(...)` 对省略的字段读取相应 `PT_SNAP_MAX_WORK_ROWS`、
`PT_SNAP_MAX_WORK_BYTES`、`PT_SNAP_MAX_SCRATCH_DB_BYTES` 环境变量,再回退默认值。
显式字段覆盖对应环境变量,即使该环境值非法。工作上限必须是正整数,不接受布尔值或
浮点数;scratch 接受整数零(禁用)或 4096 至 4294967296 字节。没有无限工作预算;
非法值抛出 core `InvalidParameterError`。

每次操作的 `query_limits` 替换整个构造配置;构造配置又覆盖环境/默认值解析。两处均
省略时,每次操作重新读取环境。若只覆盖部分字段、其余继续读取环境,应调用
`QueryLimits.resolve(...)`。`get_template_info(..., query_limits=...)` 可查看单次覆盖,
`list_capabilities()` 则反映该 analyzer 的常规有效默认配置。

`QueryService`、`ReportService` 构造函数也接受 `query_limits`;`execute_query`、
`peak_memory_report`、`event_attribution`、`memory_tree_report` 接受单次覆盖。
峰值报告步骤共用累计计数器和 deadline,独立操作使用全新计数器。可变内部
`QueryBudget` 不是公开配置接口。预算只约束数据集工作,不约束单库结果物化、RSS 或
所有 SQLite 临时空间。深分页成本、失败语义及保持不变的默认上限见
[公开配置说明](querying.md#公开数据集预算)。

## 连接生命周期与缓存所有权

analyzer 会为查询和概览复用只读 SQLite 连接。使用 `with` 可确定性释放连接,
Expand Down
12 changes: 9 additions & 3 deletions src/pt_snap_cli/api.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@
TemplateNotFoundError,
)
from pt_snap_cli.core.context_cache import ContextCache
from pt_snap_cli.core.query_limits import QueryLimits


class TemplateSummaryPayload(TypedDict):
Expand Down Expand Up @@ -70,6 +71,7 @@ def __init__(
device_id: int | None = None,
*,
context_cache: ContextCache | None = None,
query_limits: QueryLimits | None = None,
) -> None:
self._config: Config = Config()
self._db_path: Path | None = db_path
Expand All @@ -86,7 +88,7 @@ def __init__(
self._owns_context_cache: bool = context_cache is None
self._closed: bool = False
self._query_service: QueryService = QueryService(
self._focus_service, context_cache=self._context_cache
self._focus_service, context_cache=self._context_cache, query_limits=query_limits
)
self._metadata_service: ImportMetadataService = ImportMetadataService()
self._capability_service: CapabilityService = CapabilityService(
Expand Down Expand Up @@ -195,10 +197,12 @@ def list_templates(self, category: str | None = None) -> list[TemplateSummaryPay
for template in self._query_service.list_templates(category)
]

def get_template_info(self, name: str) -> dict[str, object] | None:
def get_template_info(
self, name: str, *, query_limits: QueryLimits | None = None
) -> dict[str, object] | None:
self._ensure_open()
try:
info = self._query_service.get_template_info(name)
info = self._query_service.get_template_info(name, query_limits=query_limits)
except TemplateNotFoundError:
return None

Expand All @@ -214,6 +218,7 @@ def execute_query(
exact_total: bool = False,
timeout_s: float | None = None,
slice_index: int | None = None,
query_limits: QueryLimits | None = None,
) -> QueryResultPayload:
self._ensure_open()
try:
Expand All @@ -226,6 +231,7 @@ def execute_query(
exact_total=exact_total,
timeout_s=timeout_s,
slice_index=slice_index,
query_limits=query_limits,
)
except FocusNotConfiguredError as exc:
raise RuntimeError("No database configured. Call set_focus() first.") from exc
Expand Down
Loading
Loading