diff --git a/docs/en/querying.md b/docs/en/querying.md index f71a21a..7d1c0fe 100644 --- a/docs/en/querying.md +++ b/docs/en/querying.md @@ -44,6 +44,8 @@ Aggregation and analysis. |----------|-------------| | `callstack_analysis` | Analyze callstack information | | `memory_peak` | Peak memory metrics | +| `active_blocks_at_event` | List blocks that are still active at a specific event, with optional static block inclusion | +| `allocator_gap` | Compare allocated, active, and reserved peak events and same-event gaps | ### Business Queries @@ -52,6 +54,88 @@ Domain-specific analysis. | Template | Description | |----------|-------------| | `leak_detection` | Find allocations without matching free events | +| `active_memory_callstack_at_event` | Aggregate blocks active at a specific event by allocation callstack, with static memory classified separately | + +## Peak Memory Attribution Workflow + +These additions productize the common "find the peak, then explain what was live at that moment" workflow. + +### 1. Find the peak event + +```bash +pt-snap query --template-use memory_peak +``` + +This returns peak values and the corresponding event IDs for `allocated`, `active`, and `reserved`. + +### 2. Inspect the blocks that were still active at that event + +```bash +pt-snap query --template-use active_blocks_at_event --params '{"event_id": 1234, "include_static": true}' +``` + +`active_blocks_at_event` treats a block as live at `event_id` when: + +- `allocEventId <= event_id` +- and `freeEventId = -1` or `freeEventId > event_id` + +When `include_static=true`, blocks with `allocEventId=-1 AND freeEventId=-1` are also included and labeled as `static`. + +### 3. Attribute active memory to callstacks at that event + +```bash +pt-snap query --template-use active_memory_callstack_at_event --params '{"event_id": 1234, "include_static": true, "top_n": 20}' +``` + +This query: + +- starts from the active block set at `event_id` +- joins dynamic blocks back to their allocation events in `trace_entry_` +- groups by allocation callstack +- emits static memory as a dedicated group instead of inventing a callstack + +### 4. Compare peak event gaps across metrics + +```bash +pt-snap query --template-use allocator_gap +``` + +This reports: + +- the peak event for `allocated`, `active`, and `reserved` +- whether those peaks happen at the same event +- same-event gaps such as `reserved - active` and `reserved - allocated` + +This is useful because `reserved` may peak at a different event from `active` or `allocated`, so subtracting peak values directly can be misleading. + +## Report Command + +For a higher-level summary, use the report command: + +```bash +pt-snap report peak-memory [db_path] [--device ] [--metric active|allocated|reserved] [--include-static|--exclude-static] [--limit ] [--json] +``` + +Examples: + +```bash +# Text report using the active peak event +pt-snap report peak-memory /path/to/snapshot.db + +# Report the reserved peak instead +pt-snap report peak-memory /path/to/snapshot.db --metric reserved + +# Emit machine-readable JSON +pt-snap report peak-memory /path/to/snapshot.db --json +``` + +The report command combines: + +- `memory_peak` +- `allocator_gap` +- `active_memory_callstack_at_event` + +and prints either a human-readable summary or JSON. **Example:** ```bash diff --git a/docs/zh/querying.md b/docs/zh/querying.md index 2392cc3..2bfa25f 100644 --- a/docs/zh/querying.md +++ b/docs/zh/querying.md @@ -44,6 +44,8 @@ pt-snap query [--template-use ] [--params ] [--device ] |------|------| | `callstack_analysis` | 调用栈分析 | | `memory_peak` | 峰值内存指标 | +| `active_blocks_at_event` | 查询某个事件时刻仍然活跃的 block,可选包含静态内存 | +| `allocator_gap` | 比较 allocated、active、reserved 三类峰值事件及其同事件 gap | ### Business Queries @@ -52,6 +54,88 @@ pt-snap query [--template-use ] [--params ] [--device ] | 模板 | 说明 | |------|------| | `leak_detection` | 查找未匹配释放事件的分配 | +| `active_memory_callstack_at_event` | 对某个事件时刻的活跃内存块按分配调用栈做聚合,并单独标识静态内存 | + +## 峰值内存归因工作流 + +这些新增能力把“先找到峰值,再解释峰值时刻哪些内存仍然活跃”的手工分析流程产品化了。 + +### 1. 先定位峰值事件 + +```bash +pt-snap query --template-use memory_peak +``` + +这个模板会返回 `allocated`、`active`、`reserved` 的峰值,以及各自对应的事件 ID。 + +### 2. 查看该事件时刻仍然活跃的 block + +```bash +pt-snap query --template-use active_blocks_at_event --params '{"event_id": 1234, "include_static": true}' +``` + +`active_blocks_at_event` 认为 block 在 `event_id` 时刻仍然活跃的条件是: + +- `allocEventId <= event_id` +- 且 `freeEventId = -1` 或 `freeEventId > event_id` + +当 `include_static=true` 时,还会包含 `allocEventId=-1 AND freeEventId=-1` 的 block,并标记为 `static`。 + +### 3. 对该时刻的活跃内存做调用栈归因 + +```bash +pt-snap query --template-use active_memory_callstack_at_event --params '{"event_id": 1234, "include_static": true, "top_n": 20}' +``` + +这个查询会: + +- 先从 `event_id` 时刻的活跃 block 集合开始 +- 把动态 block 回连到 `trace_entry_` 的分配事件 +- 按分配调用栈做聚合 +- 对静态内存单独分组,而不是伪造调用栈 + +### 4. 比较不同指标的峰值与 gap + +```bash +pt-snap query --template-use allocator_gap +``` + +这个模板会报告: + +- `allocated`、`active`、`reserved` 的峰值事件 +- 它们是否发生在同一个事件 +- 同一事件上的 gap,例如 `reserved - active`、`reserved - allocated` + +这样可以避免误把不同事件上的峰值直接相减,并错误解释为同一时刻的碎片或缓存 gap。 + +## Report 命令 + +如果需要更高层的摘要,可以使用: + +```bash +pt-snap report peak-memory [db_path] [--device ] [--metric active|allocated|reserved] [--include-static|--exclude-static] [--limit ] [--json] +``` + +示例: + +```bash +# 生成基于 active 峰值事件的文本报告 +pt-snap report peak-memory /path/to/snapshot.db + +# 改为查看 reserved 峰值 +pt-snap report peak-memory /path/to/snapshot.db --metric reserved + +# 输出机器可读 JSON +pt-snap report peak-memory /path/to/snapshot.db --json +``` + +这个 report 命令会组合: + +- `memory_peak` +- `allocator_gap` +- `active_memory_callstack_at_event` + +并输出人类可读摘要或 JSON。 **示例:** ```bash diff --git a/pyproject.toml b/pyproject.toml index 8ea175b..4087312 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -59,7 +59,10 @@ where = ["src"] include = ["pt_snap_cli*"] [tool.setuptools.package-data] -pt_snap_cli = ["query/templates/*.yaml"] +pt_snap_cli = [ + "query/templates/*.yaml", + "query/templates/*/*.yaml", +] [tool.black] line-length = 100 diff --git a/src/pt_snap_cli/cli.py b/src/pt_snap_cli/cli.py index 1d4f8f0..6da8807 100644 --- a/src/pt_snap_cli/cli.py +++ b/src/pt_snap_cli/cli.py @@ -5,8 +5,9 @@ import json import shlex from collections.abc import Mapping +from dataclasses import asdict from pathlib import Path -from typing import Annotated, NoReturn, cast +from typing import Annotated, Literal, NoReturn, cast import typer @@ -21,8 +22,10 @@ FocusService, InvalidCategoryError, InvalidDeviceError, + PeakMemoryReport, QueryExecutionError, QueryService, + ReportService, TemplateNotFoundError, TemplateRenderError, ) @@ -34,6 +37,8 @@ add_completion=True, context_settings={"help_option_names": ["-h", "--help"]}, ) +report_app = typer.Typer(help="Generate memory analysis reports") +app.add_typer(report_app, name="report") def _focus_service() -> FocusService: @@ -356,6 +361,136 @@ def query_database( raise typer.Exit(1) from None +@report_app.command("peak-memory") +def report_peak_memory( + db_path: Annotated[ + Path | None, typer.Argument(help="Path to database file (optional if configured)") + ] = None, + device: Annotated[ + int | None, + typer.Option( + "--device", + "-d", + help="Device ID to report", + autocompletion=complete_device_ids, + ), + ] = None, + metric: Annotated[ + Literal["active", "allocated", "reserved"], + typer.Option(help="Peak metric to report: active, allocated, or reserved"), + ] = "active", + include_static: Annotated[ + bool, + typer.Option("--include-static/--exclude-static", help="Include static memory group"), + ] = True, + limit: Annotated[int, typer.Option("--limit", "-n", help="Maximum callstack groups")] = 20, + json_output: Annotated[bool, typer.Option("--json", help="Emit machine-readable JSON")] = False, +) -> None: + """Generate a peak memory attribution report.""" + focus_service = _focus_service() + report_service = ReportService(focus_service) + try: + report = report_service.peak_memory_report( + db_path=db_path, + device_id=device, + metric=metric, + include_static=include_static, + limit=limit, + ) + except ValueError as e: + _error(str(e)) + except FocusFileInvalidError as e: + _error(str(e)) + except FocusNotConfiguredError: + typer.secho( + "Error: No database path specified and no database configured.", fg=typer.colors.RED + ) + typer.echo( + "Use 'pt-snap focus ' to set a project database, or provide db_path argument." + ) + raise typer.Exit(1) from None + except DatabaseMissingError: + try: + state = focus_service.get_focus(explicit_db_path=db_path, explicit_device_id=device) + except FocusFileInvalidError: + state = None + source = state.source if state is not None else "configured" + path = state.db_path if state is not None else db_path + typer.secho(f"Error: Database from {source} focus not found: {path}", fg=typer.colors.RED) + if state is not None and state.focus_file: + typer.echo(f"Focus file: {state.focus_file}") + typer.echo( + "Use 'pt-snap focus ' to set a new project database, or provide db_path argument." + ) + raise typer.Exit(1) from None + except InvalidDeviceError as e: + _error(str(e)) + except ( + TemplateNotFoundError, + TemplateRenderError, + QueryExecutionError, + DatabaseSchemaError, + ) as e: + typer.secho(f"Error generating report: {e}", fg=typer.colors.RED) + raise typer.Exit(1) from None + + if json_output: + typer.echo(json.dumps(asdict(report), indent=2)) + return + + _print_peak_memory_report(report) + + +def _print_peak_memory_report(report: PeakMemoryReport) -> None: + typer.secho("Peak memory report", fg=typer.colors.GREEN, bold=True) + typer.echo(f"Device: {report.device_id}") + typer.echo(f"Metric: {report.metric}") + typer.echo(f"Event ID: {report.event_id}") + typer.echo() + + typer.secho("Peak counters:", fg=typer.colors.GREEN, bold=True) + if report.peak: + typer.echo( + f" allocated: {report.peak.get('peak_allocated')} at event {report.peak.get('peak_allocated_event_id')}" + ) + typer.echo( + f" active: {report.peak.get('peak_active')} at event {report.peak.get('peak_active_event_id')}" + ) + typer.echo( + f" reserved: {report.peak.get('peak_reserved')} at event {report.peak.get('peak_reserved_event_id')}" + ) + else: + typer.echo(" No peak counters found.") + typer.echo() + + typer.secho("Allocator gap:", fg=typer.colors.GREEN, bold=True) + if report.allocator_gap: + gap = report.allocator_gap + typer.echo( + f" active peak event: {gap.get('peak_active_event_id')} reserved-active gap={gap.get('reserved_active_gap_at_active_peak')}" + ) + typer.echo( + f" allocated peak event: {gap.get('peak_allocated_event_id')} reserved-allocated gap={gap.get('reserved_allocated_gap_at_allocated_peak')}" + ) + typer.echo( + f" reserved peak event: {gap.get('peak_reserved_event_id')} reserved-active gap={gap.get('reserved_active_gap_at_reserved_peak')}" + ) + typer.echo(f" all peaks same event: {bool(gap.get('all_peaks_same_event'))}") + else: + typer.echo(" No allocator gap data found.") + typer.echo() + + typer.secho("Active memory by callstack:", fg=typer.colors.GREEN, bold=True) + if not report.callstack_groups: + typer.echo(" No active memory callstack groups found.") + return + for index, row in enumerate(report.callstack_groups, start=1): + typer.echo( + f" [{index}] {row.get('category')} {row.get('size_bytes')} bytes, {row.get('block_count')} blocks ({row.get('percent_of_active_blocks')}%)" + ) + typer.echo(f" {row.get('callstack')}") + + @app.command("config") def show_config( clear: Annotated[bool, typer.Option("--clear", help="Clear all configuration")] = False, diff --git a/src/pt_snap_cli/core/__init__.py b/src/pt_snap_cli/core/__init__.py index 582530c..1dd1064 100644 --- a/src/pt_snap_cli/core/__init__.py +++ b/src/pt_snap_cli/core/__init__.py @@ -13,6 +13,7 @@ from pt_snap_cli.core.focus_service import FocusService from pt_snap_cli.core.models import ( FocusState, + PeakMemoryReport, QueryResult, ResolvedFocus, TemplateInfo, @@ -20,6 +21,7 @@ TemplateSummary, ) from pt_snap_cli.core.query_service import QueryService +from pt_snap_cli.core.report_service import ReportService __all__ = [ "PtSnapCoreError", @@ -38,6 +40,8 @@ "TemplateSummary", "TemplateInfo", "QueryResult", + "PeakMemoryReport", "FocusService", "QueryService", + "ReportService", ] diff --git a/src/pt_snap_cli/core/models.py b/src/pt_snap_cli/core/models.py index d25b50a..e1768a5 100644 --- a/src/pt_snap_cli/core/models.py +++ b/src/pt_snap_cli/core/models.py @@ -59,3 +59,13 @@ class QueryResult: returned: int device_id: int | None rows: list[dict[str, Any]] + + +@dataclass(frozen=True) +class PeakMemoryReport: + device_id: int | None + metric: str + event_id: int | None + peak: dict[str, Any] + allocator_gap: dict[str, Any] | None + callstack_groups: list[dict[str, Any]] diff --git a/src/pt_snap_cli/core/report_service.py b/src/pt_snap_cli/core/report_service.py new file mode 100644 index 0000000..f7dd460 --- /dev/null +++ b/src/pt_snap_cli/core/report_service.py @@ -0,0 +1,82 @@ +from __future__ import annotations + +from pathlib import Path +from typing import Literal + +from pt_snap_cli.core.focus_service import FocusService +from pt_snap_cli.core.models import PeakMemoryReport +from pt_snap_cli.core.query_service import QueryService + +PeakMetric = Literal["active", "allocated", "reserved"] + +_EVENT_ID_BY_METRIC = { + "active": "peak_active_event_id", + "allocated": "peak_allocated_event_id", + "reserved": "peak_reserved_event_id", +} + + +class ReportService: + def __init__(self, focus_service: FocusService | None = None) -> None: + self._focus_service = focus_service or FocusService() + self._query_service = QueryService(self._focus_service) + + def peak_memory_report( + self, + db_path: Path | str | None = None, + device_id: int | None = None, + metric: PeakMetric = "active", + include_static: bool = True, + limit: int = 20, + start_dir: Path | None = None, + ) -> PeakMemoryReport: + if metric not in _EVENT_ID_BY_METRIC: + raise ValueError( + f"Invalid metric '{metric}'. Must be one of: active, allocated, reserved" + ) + + peak_result = self._query_service.execute_query( + "memory_peak", + db_path=db_path, + device_id=device_id, + start_dir=start_dir, + ) + peak = peak_result.rows[0] if peak_result.rows else {} + event_id = peak.get(_EVENT_ID_BY_METRIC[metric]) + + if event_id is None: + return PeakMemoryReport( + device_id=peak_result.device_id, + metric=metric, + event_id=None, + peak=peak, + allocator_gap=None, + callstack_groups=[], + ) + + gap_result = self._query_service.execute_query( + "allocator_gap", + db_path=db_path, + device_id=device_id, + start_dir=start_dir, + ) + callstack_result = self._query_service.execute_query( + "active_memory_callstack_at_event", + params={ + "event_id": event_id, + "include_static": include_static, + "top_n": limit, + }, + db_path=db_path, + device_id=device_id, + start_dir=start_dir, + ) + + return PeakMemoryReport( + device_id=peak_result.device_id, + metric=metric, + event_id=event_id, + peak=peak, + allocator_gap=gap_result.rows[0] if gap_result.rows else None, + callstack_groups=callstack_result.rows, + ) diff --git a/src/pt_snap_cli/query/templates/business/active_memory_callstack_at_event.yaml b/src/pt_snap_cli/query/templates/business/active_memory_callstack_at_event.yaml new file mode 100644 index 0000000..778b5d2 --- /dev/null +++ b/src/pt_snap_cli/query/templates/business/active_memory_callstack_at_event.yaml @@ -0,0 +1,90 @@ +version: "1.0" +queries: + active_memory_callstack_at_event: + description: Aggregate active memory blocks at an event by allocation callstack, including static memory classification + devices: + - all + parameters: + event_id: + type: int + required: true + description: Trace event ID used as the point-in-time lifecycle boundary + include_static: + type: bool + default: true + required: false + description: Include static blocks with allocEventId=-1 and freeEventId=-1 as a separate group + min_size: + type: int + default: 0 + required: false + description: Minimum individual block size in bytes + top_n: + type: int + default: 20 + required: false + description: Maximum number of callstack groups to return (-1 for unlimited) + query: | + WITH active_blocks AS ( + SELECT + id, + size, + requestedSize, + allocEventId, + freeEventId, + CASE + WHEN allocEventId = -1 AND freeEventId = -1 THEN 'static' + ELSE 'dynamic_live_at_event' + END AS category + FROM {{ device_block_table }} + WHERE size >= {{ min_size|int }} + AND ( + {% if include_static %}(allocEventId = -1 AND freeEventId = -1) OR{% endif %} + (allocEventId != -1 AND allocEventId <= {{ event_id|int }} AND (freeEventId IS NULL OR freeEventId < 0 OR freeEventId > {{ event_id|int }})) + ) + ), grouped AS ( + SELECT + CASE + WHEN b.category = 'static' THEN '[static] allocEventId=-1, freeEventId=-1' + ELSE COALESCE(t.callstack, '[missing callstack]') + END AS callstack, + b.category, + COUNT(*) AS block_count, + SUM(b.size) AS size_bytes, + ROUND(SUM(b.size) / 1073741824.0, 6) AS size_gib, + SUM(b.requestedSize) AS requested_bytes, + ROUND(SUM(b.requestedSize) / 1073741824.0, 6) AS requested_gib + FROM active_blocks b + LEFT JOIN {{ device_trace_table }} t + ON b.allocEventId = t.id + GROUP BY callstack, b.category + ) + SELECT + callstack, + category, + block_count, + size_bytes, + size_gib, + requested_bytes, + requested_gib, + ROUND(size_bytes * 100.0 / NULLIF(SUM(size_bytes) OVER (), 0), 4) AS percent_of_active_blocks + FROM grouped + ORDER BY size_bytes DESC, block_count DESC, callstack ASC + LIMIT {{ top_n|int }} + output_schema: + - column: callstack + type: str + - column: category + type: str + - column: block_count + type: int + - column: size_bytes + type: int + - column: size_gib + type: float + - column: requested_bytes + type: int + - column: requested_gib + type: float + - column: percent_of_active_blocks + type: float diff --git a/src/pt_snap_cli/query/templates/statistical/active_blocks_at_event.yaml b/src/pt_snap_cli/query/templates/statistical/active_blocks_at_event.yaml new file mode 100644 index 0000000..29eca24 --- /dev/null +++ b/src/pt_snap_cli/query/templates/statistical/active_blocks_at_event.yaml @@ -0,0 +1,91 @@ +version: "1.0" +queries: + active_blocks_at_event: + description: Query memory blocks that are active at a specific trace event + devices: + - all + parameters: + event_id: + type: int + required: true + description: Trace event ID used as the point-in-time lifecycle boundary + include_static: + type: bool + default: true + required: false + description: Include static blocks with allocEventId=-1 and freeEventId=-1 + min_size: + type: int + default: 0 + required: false + description: Minimum block size in bytes + order_by: + type: str + default: "size" + required: false + description: Sort key (size, requestedSize, allocEventId, freeEventId, id) + order_dir: + type: str + default: "DESC" + required: false + description: Sort direction (ASC or DESC) + limit: + type: int + default: -1 + required: false + description: Maximum rows to return (-1 for unlimited) + offset: + type: int + default: 0 + required: false + description: Number of rows to skip + query: | + WITH active_blocks AS ( + SELECT + id, + address, + size, + requestedSize, + state, + allocEventId, + freeEventId, + CASE + WHEN allocEventId = -1 AND freeEventId = -1 THEN 'static' + ELSE 'dynamic_live_at_event' + END AS category + FROM {{ device_block_table }} + WHERE size >= {{ min_size|int }} + AND ( + {% if include_static %}(allocEventId = -1 AND freeEventId = -1) OR{% endif %} + (allocEventId != -1 AND allocEventId <= {{ event_id|int }} AND (freeEventId IS NULL OR freeEventId < 0 OR freeEventId > {{ event_id|int }})) + ) + ) + SELECT * + FROM active_blocks + ORDER BY + {% if order_by == 'requestedSize' %}requestedSize + {% elif order_by == 'allocEventId' %}allocEventId + {% elif order_by == 'freeEventId' %}freeEventId + {% elif order_by == 'id' %}id + {% else %}size + {% endif %} + {% if order_dir|upper == 'ASC' %}ASC{% else %}DESC{% endif %}, id ASC + LIMIT {{ limit|int }} + {% if offset > 0 %}OFFSET {{ offset|int }}{% endif %} + output_schema: + - column: id + type: int + - column: address + type: int + - column: size + type: int + - column: requestedSize + type: int + - column: state + type: int + - column: allocEventId + type: int + - column: freeEventId + type: int + - column: category + type: str diff --git a/src/pt_snap_cli/query/templates/statistical/allocator_gap.yaml b/src/pt_snap_cli/query/templates/statistical/allocator_gap.yaml new file mode 100644 index 0000000..26cf20c --- /dev/null +++ b/src/pt_snap_cli/query/templates/statistical/allocator_gap.yaml @@ -0,0 +1,109 @@ +version: "1.0" +queries: + allocator_gap: + description: Compare allocated, active, and reserved peak events and same-event gaps + devices: + - all + parameters: + start_id: + type: int + default: null + required: false + description: Start event ID for range filtering + end_id: + type: int + default: null + required: false + description: End event ID for range filtering + query: | + WITH filtered AS ( + SELECT id, allocated, active, reserved + FROM {{ device_trace_table }} + WHERE 1=1 + {% if start_id is not none %}AND id >= {{ start_id|int }}{% endif %} + {% if end_id is not none %}AND id <= {{ end_id|int }}{% endif %} + ), peaks AS ( + SELECT + (SELECT id FROM filtered ORDER BY allocated DESC, id ASC LIMIT 1) AS peak_allocated_event_id, + (SELECT allocated FROM filtered ORDER BY allocated DESC, id ASC LIMIT 1) AS peak_allocated, + (SELECT id FROM filtered ORDER BY active DESC, id ASC LIMIT 1) AS peak_active_event_id, + (SELECT active FROM filtered ORDER BY active DESC, id ASC LIMIT 1) AS peak_active, + (SELECT id FROM filtered ORDER BY reserved DESC, id ASC LIMIT 1) AS peak_reserved_event_id, + (SELECT reserved FROM filtered ORDER BY reserved DESC, id ASC LIMIT 1) AS peak_reserved + ) + SELECT + p.peak_allocated_event_id, + p.peak_allocated, + allocated_event.active AS active_at_allocated_peak, + allocated_event.reserved AS reserved_at_allocated_peak, + allocated_event.reserved - allocated_event.active AS reserved_active_gap_at_allocated_peak, + allocated_event.reserved - allocated_event.allocated AS reserved_allocated_gap_at_allocated_peak, + p.peak_active_event_id, + p.peak_active, + active_event.allocated AS allocated_at_active_peak, + active_event.reserved AS reserved_at_active_peak, + active_event.reserved - active_event.active AS reserved_active_gap_at_active_peak, + active_event.reserved - active_event.allocated AS reserved_allocated_gap_at_active_peak, + p.peak_reserved_event_id, + p.peak_reserved, + reserved_event.allocated AS allocated_at_reserved_peak, + reserved_event.active AS active_at_reserved_peak, + reserved_event.reserved - reserved_event.active AS reserved_active_gap_at_reserved_peak, + reserved_event.reserved - reserved_event.allocated AS reserved_allocated_gap_at_reserved_peak, + CASE + WHEN p.peak_allocated_event_id = p.peak_active_event_id + AND p.peak_active_event_id = p.peak_reserved_event_id + THEN 1 ELSE 0 + END AS all_peaks_same_event, + CASE WHEN p.peak_allocated_event_id = p.peak_active_event_id THEN 1 ELSE 0 END AS allocated_active_same_event, + CASE WHEN p.peak_active_event_id = p.peak_reserved_event_id THEN 1 ELSE 0 END AS active_reserved_same_event, + CASE WHEN p.peak_allocated_event_id = p.peak_reserved_event_id THEN 1 ELSE 0 END AS allocated_reserved_same_event + FROM peaks p + LEFT JOIN filtered allocated_event ON allocated_event.id = p.peak_allocated_event_id + LEFT JOIN filtered active_event ON active_event.id = p.peak_active_event_id + LEFT JOIN filtered reserved_event ON reserved_event.id = p.peak_reserved_event_id + output_schema: + - column: peak_allocated_event_id + type: int + - column: peak_allocated + type: int + - column: active_at_allocated_peak + type: int + - column: reserved_at_allocated_peak + type: int + - column: reserved_active_gap_at_allocated_peak + type: int + - column: reserved_allocated_gap_at_allocated_peak + type: int + - column: peak_active_event_id + type: int + - column: peak_active + type: int + - column: allocated_at_active_peak + type: int + - column: reserved_at_active_peak + type: int + - column: reserved_active_gap_at_active_peak + type: int + - column: reserved_allocated_gap_at_active_peak + type: int + - column: peak_reserved_event_id + type: int + - column: peak_reserved + type: int + - column: allocated_at_reserved_peak + type: int + - column: active_at_reserved_peak + type: int + - column: reserved_active_gap_at_reserved_peak + type: int + - column: reserved_allocated_gap_at_reserved_peak + type: int + - column: all_peaks_same_event + type: int + - column: allocated_active_same_event + type: int + - column: active_reserved_same_event + type: int + - column: allocated_reserved_same_event + type: int diff --git a/tests/core/test_report_service.py b/tests/core/test_report_service.py new file mode 100644 index 0000000..558591e --- /dev/null +++ b/tests/core/test_report_service.py @@ -0,0 +1,147 @@ +"""Tests for peak memory report service.""" + +import sqlite3 +from pathlib import Path + +import pytest + +from pt_snap_cli.core.report_service import ReportService +from pt_snap_cli.query.registry import QueryRegistry, _load_all_templates + + +@pytest.fixture(autouse=True) +def _reload_query_templates(): + QueryRegistry.reset() + _load_all_templates() + + +@pytest.fixture +def report_db(tmp_path: Path) -> Path: + db_path = tmp_path / "report.db" + conn = sqlite3.connect(str(db_path)) + conn.execute(""" + CREATE TABLE dictionary ( + `table` TEXT, + `column` TEXT, + `key` TEXT, + `value` TEXT + ) + """) + conn.execute(""" + CREATE TABLE trace_entry_0 ( + id INTEGER PRIMARY KEY, + action INTEGER, + address INTEGER, + size INTEGER, + stream INTEGER, + allocated INTEGER, + active INTEGER, + reserved INTEGER, + callstack TEXT + ) + """) + conn.execute(""" + CREATE TABLE block_0 ( + id INTEGER PRIMARY KEY, + address INTEGER, + size INTEGER, + requestedSize INTEGER, + state INTEGER, + allocEventId INTEGER, + freeEventId INTEGER + ) + """) + conn.executemany( + """ + INSERT INTO trace_entry_0 + (id, action, address, size, stream, allocated, active, reserved, callstack) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?) + """, + [ + (1, 2, 0x1000, 1024, 0, 1024, 1024, 4096, "train.py:10"), + (2, 2, 0x2000, 2048, 0, 3072, 3072, 4096, "block.py:20"), + (3, 3, 0x2000, 2048, 0, 1024, 1024, 8192, "free.py:30"), + (4, 2, 0x4000, 4096, 0, 5120, 5120, 8192, "after.py:40"), + (5, 2, 0x5000, 512, 0, 5632, 5632, 8192, None), + ], + ) + conn.executemany( + """ + INSERT INTO block_0 + (id, address, size, requestedSize, state, allocEventId, freeEventId) + VALUES (?, ?, ?, ?, ?, ?, ?) + """, + [ + (-10, 0xA000, 8192, 8000, 1, -1, -1), + (1, 0x1000, 1024, 1000, 1, 1, -1), + (2, 0x2000, 2048, 2000, 0, 2, 3), + (4, 0x4000, 4096, 4000, 1, 4, -1), + (5, 0x5000, 512, 500, 1, 5, -1), + ], + ) + conn.commit() + conn.close() + return db_path + + +def test_peak_memory_report_selects_active_peak_event(report_db: Path) -> None: + report = ReportService().peak_memory_report(report_db, metric="active") + + assert report.device_id == 0 + assert report.metric == "active" + assert report.event_id == 5 + assert report.peak["peak_active"] == 5632 + assert report.allocator_gap is not None + assert report.allocator_gap["peak_reserved_event_id"] == 3 + assert report.callstack_groups[0]["callstack"] == "[static] allocEventId=-1, freeEventId=-1" + + +def test_peak_memory_report_can_exclude_static(report_db: Path) -> None: + report = ReportService().peak_memory_report(report_db, include_static=False) + + assert "[static] allocEventId=-1, freeEventId=-1" not in { + row["callstack"] for row in report.callstack_groups + } + + +def test_peak_memory_report_rejects_invalid_metric(report_db: Path) -> None: + with pytest.raises(ValueError, match="Invalid metric"): + ReportService().peak_memory_report(report_db, metric="invalid") # type: ignore[arg-type] + + +def test_peak_memory_report_handles_empty_trace(tmp_path: Path) -> None: + db_path = tmp_path / "empty.db" + conn = sqlite3.connect(str(db_path)) + conn.execute("CREATE TABLE dictionary (`table` TEXT, `column` TEXT, `key` TEXT, `value` TEXT)") + conn.execute(""" + CREATE TABLE trace_entry_0 ( + id INTEGER PRIMARY KEY, + action INTEGER, + address INTEGER, + size INTEGER, + stream INTEGER, + allocated INTEGER, + active INTEGER, + reserved INTEGER, + callstack TEXT + ) + """) + conn.execute(""" + CREATE TABLE block_0 ( + id INTEGER PRIMARY KEY, + address INTEGER, + size INTEGER, + requestedSize INTEGER, + state INTEGER, + allocEventId INTEGER, + freeEventId INTEGER + ) + """) + conn.commit() + conn.close() + + report = ReportService().peak_memory_report(db_path) + + assert report.event_id is None + assert report.allocator_gap is None + assert report.callstack_groups == [] diff --git a/tests/query/test_peak_memory_templates.py b/tests/query/test_peak_memory_templates.py new file mode 100644 index 0000000..e3e9b6a --- /dev/null +++ b/tests/query/test_peak_memory_templates.py @@ -0,0 +1,237 @@ +"""Tests for peak memory attribution query templates.""" + +import sqlite3 +from pathlib import Path + +import pytest + +from pt_snap_cli.context import Context +from pt_snap_cli.query.executor import QueryExecutor +from pt_snap_cli.query.registry import ( + QueryRegistry, + _load_all_templates, + get_query, + list_by_category, +) + + +@pytest.fixture(autouse=True) +def _reload_query_templates(): + QueryRegistry.reset() + _load_all_templates() + + +@pytest.fixture +def peak_memory_db(tmp_path: Path) -> Path: + db_path = tmp_path / "peak_memory.db" + conn = sqlite3.connect(str(db_path)) + conn.execute(""" + CREATE TABLE dictionary ( + `table` TEXT, + `column` TEXT, + `key` TEXT, + `value` TEXT + ) + """) + conn.execute(""" + CREATE TABLE trace_entry_0 ( + id INTEGER PRIMARY KEY, + action INTEGER, + address INTEGER, + size INTEGER, + stream INTEGER, + allocated INTEGER, + active INTEGER, + reserved INTEGER, + callstack TEXT + ) + """) + conn.execute(""" + CREATE TABLE block_0 ( + id INTEGER PRIMARY KEY, + address INTEGER, + size INTEGER, + requestedSize INTEGER, + state INTEGER, + allocEventId INTEGER, + freeEventId INTEGER + ) + """) + + conn.executemany( + """ + INSERT INTO trace_entry_0 + (id, action, address, size, stream, allocated, active, reserved, callstack) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?) + """, + [ + (1, 2, 0x1000, 1024, 0, 1024, 1024, 4096, "train.py:10"), + (2, 2, 0x2000, 2048, 0, 3072, 3072, 4096, "freed.py:20"), + (3, 3, 0x2000, 2048, 0, 1024, 1024, 8192, "free.py:30"), + (4, 2, 0x4000, 4096, 0, 5120, 5120, 8192, "after.py:40"), + (5, 2, 0x5000, 512, 0, 5632, 5632, 8192, None), + ], + ) + conn.executemany( + """ + INSERT INTO block_0 + (id, address, size, requestedSize, state, allocEventId, freeEventId) + VALUES (?, ?, ?, ?, ?, ?, ?) + """, + [ + (-10, 0xA000, 8192, 8000, 1, -1, -1), + (1, 0x1000, 1024, 1000, 1, 1, -1), + (2, 0x2000, 2048, 2000, 0, 2, 3), + (4, 0x4000, 4096, 4000, 1, 4, -1), + (5, 0x5000, 512, 500, 1, 5, -1), + ], + ) + conn.commit() + conn.close() + return db_path + + +def _execute_template(db_path: Path, name: str, params: dict) -> list[dict]: + context = Context(db_path) + executor = QueryExecutor(context) + return executor.execute_template(name, params=params, device_id=0) + + +def test_new_templates_are_registered_in_correct_categories() -> None: + assert "active_blocks_at_event" in list_by_category("statistical") + assert "allocator_gap" in list_by_category("statistical") + assert "active_memory_callstack_at_event" in list_by_category("business") + + assert get_query("active_blocks_at_event") is not None + assert get_query("allocator_gap") is not None + assert get_query("active_memory_callstack_at_event") is not None + + +def test_active_blocks_at_event_includes_static_when_enabled(peak_memory_db: Path) -> None: + rows = _execute_template( + peak_memory_db, + "active_blocks_at_event", + {"event_id": 3, "include_static": True}, + ) + + assert [row["id"] for row in rows] == [-10, 1] + assert rows[0]["category"] == "static" + assert rows[1]["category"] == "dynamic_live_at_event" + + +def test_active_blocks_at_event_excludes_static_when_disabled(peak_memory_db: Path) -> None: + rows = _execute_template( + peak_memory_db, + "active_blocks_at_event", + {"event_id": 3, "include_static": False}, + ) + + assert [row["id"] for row in rows] == [1] + + +def test_active_blocks_at_event_applies_event_boundary_and_size_filter( + peak_memory_db: Path, +) -> None: + rows = _execute_template( + peak_memory_db, + "active_blocks_at_event", + { + "event_id": 5, + "include_static": False, + "min_size": 1000, + "order_by": "id", + "order_dir": "ASC", + }, + ) + + assert [row["id"] for row in rows] == [1, 4] + + +def test_active_blocks_at_event_treats_null_free_event_as_live(peak_memory_db: Path) -> None: + conn = sqlite3.connect(str(peak_memory_db)) + conn.execute( + """ + INSERT INTO block_0 + (id, address, size, requestedSize, state, allocEventId, freeEventId) + VALUES (?, ?, ?, ?, ?, ?, ?) + """, + (6, 0x6000, 1536, 1500, 1, 2, None), + ) + conn.commit() + conn.close() + + rows = _execute_template( + peak_memory_db, + "active_blocks_at_event", + {"event_id": 3, "include_static": False, "order_by": "id", "order_dir": "ASC"}, + ) + + assert [row["id"] for row in rows] == [1, 6] + + +def test_active_memory_callstack_at_event_treats_null_free_event_as_live( + peak_memory_db: Path, +) -> None: + conn = sqlite3.connect(str(peak_memory_db)) + conn.execute( + """ + INSERT INTO block_0 + (id, address, size, requestedSize, state, allocEventId, freeEventId) + VALUES (?, ?, ?, ?, ?, ?, ?) + """, + (6, 0x6000, 1536, 1500, 1, 2, None), + ) + conn.commit() + conn.close() + + rows = _execute_template( + peak_memory_db, + "active_memory_callstack_at_event", + {"event_id": 3, "include_static": False, "top_n": -1}, + ) + + by_callstack = {row["callstack"]: row for row in rows} + assert by_callstack["freed.py:20"]["size_bytes"] == 1536 + assert by_callstack["freed.py:20"]["block_count"] == 1 + + +def test_active_memory_callstack_at_event_groups_dynamic_static_and_missing_callstacks( + peak_memory_db: Path, +) -> None: + rows = _execute_template( + peak_memory_db, + "active_memory_callstack_at_event", + {"event_id": 5, "include_static": True, "top_n": -1}, + ) + + by_callstack = {row["callstack"]: row for row in rows} + assert by_callstack["[static] allocEventId=-1, freeEventId=-1"]["category"] == "static" + assert by_callstack["[static] allocEventId=-1, freeEventId=-1"]["size_bytes"] == 8192 + assert by_callstack["after.py:40"]["category"] == "dynamic_live_at_event" + assert by_callstack["after.py:40"]["size_bytes"] == 4096 + assert by_callstack["train.py:10"]["block_count"] == 1 + assert by_callstack["[missing callstack]"]["requested_bytes"] == 500 + + +def test_active_memory_callstack_at_event_can_exclude_static(peak_memory_db: Path) -> None: + rows = _execute_template( + peak_memory_db, + "active_memory_callstack_at_event", + {"event_id": 5, "include_static": False, "top_n": -1}, + ) + + assert "[static] allocEventId=-1, freeEventId=-1" not in {row["callstack"] for row in rows} + + +def test_allocator_gap_reports_peak_events_and_same_event_gaps(peak_memory_db: Path) -> None: + rows = _execute_template(peak_memory_db, "allocator_gap", {}) + + assert len(rows) == 1 + row = rows[0] + assert row["peak_allocated_event_id"] == 5 + assert row["peak_active_event_id"] == 5 + assert row["peak_reserved_event_id"] == 3 + assert row["allocated_active_same_event"] == 1 + assert row["active_reserved_same_event"] == 0 + assert row["reserved_active_gap_at_active_peak"] == 2560 + assert row["reserved_active_gap_at_reserved_peak"] == 7168 diff --git a/tests/test_cli.py b/tests/test_cli.py index b946139..b206f8e 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -74,6 +74,72 @@ def create_sample_db(db_path: Path, size: int = 1024) -> Path: return db_path +def create_peak_report_db(db_path: Path) -> Path: + """Create a memory snapshot database for peak report CLI tests.""" + conn = sqlite3.connect(str(db_path)) + conn.execute(""" + CREATE TABLE dictionary ( + `table` TEXT, + `column` TEXT, + `key` TEXT, + `value` TEXT + ) + """) + conn.execute(""" + CREATE TABLE trace_entry_0 ( + id INTEGER PRIMARY KEY, + action INTEGER, + address INTEGER, + size INTEGER, + stream INTEGER, + allocated INTEGER, + active INTEGER, + reserved INTEGER, + callstack TEXT + ) + """) + conn.execute(""" + CREATE TABLE block_0 ( + id INTEGER PRIMARY KEY, + address INTEGER, + size INTEGER, + requestedSize INTEGER, + state INTEGER, + allocEventId INTEGER, + freeEventId INTEGER + ) + """) + conn.executemany( + """ + INSERT INTO trace_entry_0 + (id, action, address, size, stream, allocated, active, reserved, callstack) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?) + """, + [ + (1, 2, 0x1000, 1024, 0, 1024, 1024, 4096, "train.py:10"), + (2, 2, 0x2000, 2048, 0, 3072, 3072, 4096, "block.py:20"), + (3, 3, 0x2000, 2048, 0, 1024, 1024, 8192, "free.py:30"), + (4, 2, 0x4000, 4096, 0, 5120, 5120, 8192, "after.py:40"), + ], + ) + conn.executemany( + """ + INSERT INTO block_0 + (id, address, size, requestedSize, state, allocEventId, freeEventId) + VALUES (?, ?, ?, ?, ?, ?, ?) + """, + [ + (-10, 0xA000, 8192, 8000, 1, -1, -1), + (1, 0x1000, 1024, 1000, 1, 1, -1), + (2, 0x2000, 2048, 2000, 0, 2, 3), + (4, 0x4000, 4096, 4000, 1, 4, -1), + ], + ) + conn.commit() + conn.close() + return db_path + + @pytest.fixture def sample_db(tmp_path: Path) -> Path: """Create a sample SQLite database for testing.""" @@ -730,10 +796,9 @@ def test_safe_call_catches_comp_keyerror(self) -> None: from pt_snap_cli.cli import _safe_call - with patch("sys.argv", ["pt-snap"]): - with patch("click.core.Command.parse_args", side_effect=KeyError("COMP_WORDS")): - exit_code = _safe_call() - assert exit_code == 1 + with patch("pt_snap_cli.cli.app", side_effect=KeyError("COMP_WORDS")): + exit_code = _safe_call() + assert exit_code == 1 def test_safe_call_reraises_unrelated_keyerror(self) -> None: """Test _safe_call re-raises KeyError not related to shell completion.""" @@ -741,7 +806,78 @@ def test_safe_call_reraises_unrelated_keyerror(self) -> None: from pt_snap_cli.cli import _safe_call - with patch("sys.argv", ["pt-snap"]): - with patch("click.core.Command.parse_args", side_effect=KeyError("some_other_key")): - with pytest.raises(KeyError, match="some_other_key"): - _safe_call() + with patch("pt_snap_cli.cli.app", side_effect=KeyError("some_other_key")): + with pytest.raises(KeyError, match="some_other_key"): + _safe_call() + + +class TestReportCommand: + """Test report subcommands.""" + + def test_report_help(self) -> None: + result = runner.invoke(app, ["report", "--help"]) + assert result.exit_code == 0 + assert "Generate memory analysis reports" in result.stdout + + def test_peak_memory_report_text_output(self, tmp_path: Path) -> None: + report_db = create_peak_report_db(tmp_path / "report.db") + + result = runner.invoke(app, ["report", "peak-memory", str(report_db)]) + + assert result.exit_code == 0 + assert "Peak memory report" in result.stdout + assert "Metric: active" in result.stdout + assert "Allocator gap:" in result.stdout + assert "Active memory by callstack:" in result.stdout + assert "[static] allocEventId=-1, freeEventId=-1" in result.stdout + + def test_peak_memory_report_json_output(self, tmp_path: Path) -> None: + report_db = create_peak_report_db(tmp_path / "report.db") + + result = runner.invoke(app, ["report", "peak-memory", str(report_db), "--json"]) + + assert result.exit_code == 0 + payload = json.loads(result.stdout) + assert payload["metric"] == "active" + assert payload["event_id"] == 4 + assert payload["peak"]["peak_active_event_id"] == 4 + assert payload["allocator_gap"]["peak_reserved_event_id"] == 3 + assert isinstance(payload["callstack_groups"], list) + + def test_peak_memory_report_invalid_metric(self, tmp_path: Path) -> None: + report_db = create_peak_report_db(tmp_path / "report.db") + + result = runner.invoke( + app, + ["report", "peak-memory", str(report_db), "--metric", "invalid"], + ) + + assert result.exit_code == 2 + assert "Invalid value" in result.stdout or "Invalid value" in result.stderr + + def test_peak_memory_report_exclude_static(self, tmp_path: Path) -> None: + report_db = create_peak_report_db(tmp_path / "report.db") + + result = runner.invoke( + app, + ["report", "peak-memory", str(report_db), "--exclude-static", "--json"], + ) + + assert result.exit_code == 0 + payload = json.loads(result.stdout) + assert "[static] allocEventId=-1, freeEventId=-1" not in { + row["callstack"] for row in payload["callstack_groups"] + } + + def test_peak_memory_report_configured_db_not_found(self, tmp_path: Path) -> None: + config_dir = tmp_path / ".config" / "pt-snap-cli" + config_dir.mkdir(parents=True) + missing_db = tmp_path / "missing.db" + config_file = config_dir / "config.json" + config_file.write_text(json.dumps({"db_path": str(missing_db)})) + + result = runner.invoke(app, ["report", "peak-memory"]) + + assert result.exit_code == 1 + assert "Database from global focus not found" in result.stdout + assert str(missing_db) in result.stdout