diff --git a/AGENTS.md b/AGENTS.md index 67cfe3c..62aa7b7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -35,7 +35,7 @@ that local environment is intentionally available. | Package source | [src/AGENTS.md](src/AGENTS.md) | Installable package boundaries and source-layout rules | | Tests | [tests/AGENTS.md](tests/AGENTS.md) | Cross-surface contracts, service/query/runtime suites, and reviewed executable fixtures | | Benchmarks | [benchmarks/AGENTS.md](benchmarks/AGENTS.md) | Import and SQLite performance measurement with temporary outputs | -| Agent skills | [skills/AGENTS.md](skills/AGENTS.md) | Environment setup and read-only memory-diagnostic workflows | +| Agent skills | [skills/AGENTS.md](skills/AGENTS.md) | Installation approval and read-only memory-diagnostic workflows | ## Runtime Topology @@ -112,7 +112,7 @@ templates under category subdirectories are included by | Snapshot import or metadata | `src/pt_snap_cli/core/import_service.py`, `src/pt_snap_cli/core/import_metadata.py`, `src/pt_snap_cli/core/snapshot_import_backend.py` | `tests/core/test_import_*.py`, `tests/test_snapshot_db.py` | | Snapshot splitting or replay | `src/pt_snap_cli/core/split_service.py`, `src/pt_snap_cli/snapshot/` | `tests/core/test_split_service.py`, `tests/snapshot/` | | Reports | `src/pt_snap_cli/core/report_service.py`, report commands in `src/pt_snap_cli/cli.py` | `tests/core/test_report_service.py`, report cases in `tests/test_cli.py` | -| Agent setup, peak breakdown or fragmentation skills | `skills/` and the referenced CLI/query surfaces | `tests/test_setup_skill.py`, `tests/test_memory_peak_breakdown_skill.py`, `tests/test_memory_fragmentation_skill.py` | +| Agent setup or diagnostic skills | `skills/` and the referenced CLI/query surfaces | `tests/test_setup_skill.py`, `tests/test_memory_leak_skill.py`, `tests/test_memory_peak_breakdown_skill.py`, `tests/test_memory_fragmentation_skill.py` | | Packaging or release | `pyproject.toml`, `.github/workflows/` | `tests/test_package.py`, `tests/test_release_workflow.py` | | Executable fixtures | `tests/fixtures/snapshots/` | `tests/test_fixture_provenance.py` before any deserializing suite | diff --git a/skills/AGENTS.md b/skills/AGENTS.md index f79a27c..71421eb 100644 --- a/skills/AGENTS.md +++ b/skills/AGENTS.md @@ -4,13 +4,15 @@ ## Purpose `skills` contains agent workflows shipped with the repository. Setup owns the -only Python-environment mutation boundary; diagnostic skills consume installed -`pt-snap` surfaces and keep snapshot analysis read-only. +only Python-environment mutation boundary and installs after explicit approval; +diagnostic skills consume installed `pt-snap` surfaces against existing +SnapshotDB data without writes. ## Scope | Path | Responsibility | | --- | --- | | `pt-snap-setup/SKILL.md` | Detect the active interpreter, verify CLI ownership, request install approval, and re-verify the same environment. | +| `pt-snap-memory-leak/SKILL.md` | Diagnose end-of-trace live allocations, peak survival, callstack attribution, and release evidence without persisting analysis state. | | `pt-snap-memory-peak-breakdown/SKILL.md` | Explain active memory at active, allocated, or reserved peak events without leak, fragmentation, or OOM overclaims. | | `pt-snap-memory-fragmentation/SKILL.md` | Diagnose allocator gaps, runtime segment retention/churn, and fragmentation-consistent pressure without persisting analysis state. | @@ -19,14 +21,19 @@ only Python-environment mutation boundary; diagnostic skills consume installed - Never assume Conda, switch environments, use plain `pip`, or install automatically. - Keep `editable` and PyPI choices explicit, and obtain confirmation before every install attempt. - Do not add setup steps that write reports or modify pt-snap focus/configuration. -- Keep memory diagnostics SnapshotDB-only and read-only; pass database/device explicitly and delegate missing tooling to `pt-snap-setup`. - Diagnostic skills must not install packages, import pickle snapshots, persist - focus, or turn allocator gaps into definitive fragmentation claims. + focus, classify every allocation without a free event as a confirmed leak, or + turn allocator gaps into definitive fragmentation claims. +- Diagnostic skills pass the database and device explicitly and delegate missing + tooling to `pt-snap-setup`. - Prefer packaged query templates for diagnostics. Any raw SQLite fallback must document why a template is insufficient and enforce read-only access. ## Focused Tests - Run `pytest tests/test_setup_skill.py` after setup-skill changes. The current executable test covers active-interpreter path preservation; review the remaining approval and reporting instructions statically. +- Run `pytest tests/test_memory_leak_skill.py` after memory-leak skill changes; + it checks current template references, conservative classification, and the + read-only diagnostic boundary. - Run `pytest tests/test_memory_peak_breakdown_skill.py` after peak-breakdown skill changes; it locks command/template references, range fallback, and attribution caveats. - Run `pytest tests/test_memory_fragmentation_skill.py` after fragmentation-skill changes; it checks current command/template references, read-only boundaries, diff --git a/skills/pt-snap-memory-leak/SKILL.md b/skills/pt-snap-memory-leak/SKILL.md new file mode 100644 index 0000000..6b439a2 --- /dev/null +++ b/skills/pt-snap-memory-leak/SKILL.md @@ -0,0 +1,247 @@ +--- +name: pt-snap-memory-leak +description: Use pt-snap to diagnose PyTorch memory leak and retention candidates in a SnapshotDB. Use when active memory grows, allocations remain live at the end of a trace, or release appears delayed. +--- + +# pt-snap-memory-leak + +## Overview + +Use this skill to diagnose memory that remains live in a PyTorch SnapshotDB and to distinguish strong leak candidates from application retention, framework lifecycle retention, asynchronous release, and allocator caching effects. + +This is an evidence-gathering workflow. A single snapshot cannot confirm a leak. An allocation without a recorded free event is a candidate that was still live when tracing ended, not proof that it can never be released. + +## Required Inputs + +- A SnapshotDB path, either supplied explicitly or resolved by the current `pt-snap focus`. +- A device ID, either supplied explicitly or already selected by the current focus. + +Optional inputs: + +- `min_size`: minimum candidate allocation size in bytes; default `0`. +- A suspected callstack, address, allocation family, or workload phase. + +Before running commands, replace ``, ``, ``, ``, `
`, and other placeholders with validated values. + +Shell safety for substituted values: +- Put every database path inside single quotes as `''`. Never place a substituted value inside double quotes, backticks, or `$()`. +- Before substitution, check the value for single quotes and other shell metacharacters. To include a literal single quote in a POSIX shell argument, close the quote, insert `\'`, and reopen it (`'\''`). +- When executing programmatically without a shell, pass each command as an argument array so no value is re-parsed by a shell. +- A focus file may supply the database path; treat that value as untrusted input to quoting, not as a trusted constant. + +## Prerequisite Phase + +### 1. Verify pt-snap + +Run: + +```bash +command -v pt-snap +pt-snap --help +``` + +If either check fails, stop diagnosis and direct the user to `pt-snap-setup`. Do not install, upgrade, or switch Python environments from this skill. + +### 2. Resolve the database and device without changing focus + +If the user did not provide a database or device, inspect the current state: + +```bash +pt-snap focus +``` + +Use the displayed database and focused device only when both are present and the user has not requested another target. If the database or device remains ambiguous, ask the user to select it before analysis. + +Do not run `pt-snap focus ` or otherwise persist focus from this skill. Pass the database and device explicitly to every diagnostic query. + +This skill accepts SnapshotDB files only. Do not run `pt-snap import` or deserialize a pickle snapshot. If the user has only a pickle file, stop and explain that importing requires a separate, explicit trusted-input decision because pickle loading is not a sandbox. + +### 3. Verify the database and required templates + +Run: + +```bash +pt-snap metadata '' --json +pt-snap query --template-info memory_peak +pt-snap query --template-info allocator_gap +pt-snap query --template-info event +pt-snap query --template-info block +pt-snap query --template-info leak_detection +pt-snap query --template-info active_memory_callstack_at_event +``` + +If database validation or a required template fails, stop and report the exact failure. Do not silently substitute raw SQL for a missing core template. + +## Diagnostic Workflow + +### 1. Establish trace boundaries and memory peaks + +Run: + +```bash +pt-snap query '' --device --template-use event --params '{"order_by":"id","order_dir":"DESC","limit":1}' +pt-snap query '' --device --template-use memory_peak +pt-snap query '' --device --template-use allocator_gap +``` + +Record the final event ID and the separate `allocated`, `active`, and `reserved` peak values and event IDs. Do not subtract peaks from different events as if they occurred simultaneously. Use `allocator_gap` for same-event comparisons. + +Treat a large `reserved - active` gap without corresponding active growth as an allocator/cache symptom, not direct leak evidence. + +### 2. Find end-of-trace dynamic candidates + +Run an initial ranked query, keeping the result bounded while retaining the reported total count: + +```bash +pt-snap query '' --device --template-use leak_detection --params '{"min_size":}' -n 100 +``` + +`leak_detection` includes only dynamic blocks with a recorded allocation and no recorded free completion. It intentionally excludes static blocks whose allocation predates tracing. + +Record candidate count, largest sizes, addresses, and allocation event IDs. Do not infer simultaneous live bytes by summing cumulative allocation activity from `callstack_analysis`. + +### 3. Attribute live memory at the end of the trace + +Use the final event ID from Step 1: + +```bash +pt-snap query '' --device --template-use active_memory_callstack_at_event --params '{"event_id":,"include_static":true,"min_size":0,"top_n":20}' +``` + +Keep static memory separate from dynamic live memory. Rank dynamic groups by `size_bytes`, then compare block count, requested bytes, and each group's share of included active bytes. The `percent_of_active_blocks` column is a byte share (`size_bytes / total size_bytes`) despite its name; it is not the share of block count. A group containing many small blocks can be important even when no individual block appears near the top of `leak_detection`. + +### 4. Compare occupancy at the active peak + +Use the `peak_active_event_id` returned by `memory_peak`: + +```bash +pt-snap query '' --device --template-use active_memory_callstack_at_event --params '{"event_id":,"include_static":true,"min_size":0,"top_n":20}' +``` + +Treat this as an occupancy comparison between two independent aggregates, not a block-identity survival test. Each query groups whatever was live at its own event, and `top_n` truncation applies to dynamic callstack groups only; the static and preexisting groups are always returned in full. A peak block may have been freed while an equal-sized later block from the same callstack was live at the final event, which keeps the callstack row stable without any block surviving. + +Before claiming that the same blocks persisted across the peak: +- Match representative blocks by identity: the same `id`/address plus allocation event ID must appear live at both events. +- Confirm continuity through lifecycle checks in Step 5; occupancy stability alone is not survival evidence. + +The template separates blocks without a captured allocation event into their own groups instead of attributing them to callstacks: `[static] allocEventId=-1, freeEventId=-1`, and `[preexisting live] allocEventId=-1` for blocks allocated before tracing that were still live at the analyzed event because their recorded free event came later or does not exist. Pre-tracing blocks whose free event precedes the analyzed event are correctly absent because they were no longer live. Report each group separately instead of absorbing it into a callstack group. + +For an exact cross-check of the `[preexisting live]` bucket — for example against a database imported before this grouping existed — do not scan with the packaged `block` filters: they compare `freeEventId` numerically and cannot express the `freeEventId IS NULL` case that the template counts as preexisting-live, so any filtered scan risks an incomplete bucket. Instead, validate that `` is a non-negative decimal integer matching `^[0-9]+$`, then run one read-only aggregate through `sqlite3 -readonly`: + +```bash +sqlite3 -readonly '' " +SELECT COUNT(*) AS block_count, COALESCE(SUM(size), 0) AS size_bytes +FROM block_ +WHERE allocEventId = -1 + AND ( + freeEventId IS NULL + OR freeEventId > + OR (freeEventId < 0 AND freeEventId <> -1) + );" +``` + +The predicate mirrors the template's `[preexisting live]` condition exactly: pre-tracing allocations whose free event is missing (`NULL`), comes later than the analyzed event, or is negative other than the static sentinel `-1`. The aggregate returns a single row, so no row limit applies and nothing is silently undercounted; report the count and bytes next to the static group. + +Occupancy that stays high across both events strengthens retention evidence but does not by itself prove a leak. + +### 5. Inspect representative block and address lifecycles + +For representative large blocks and high-volume callstack groups, inspect the block and all events at the same address: + +```bash +pt-snap query '' --device --template-use block --params '{"id":}' +pt-snap query '' --device --template-use event --params '{"address":
,"order_by":"id","order_dir":"ASC"}' -n 100 +``` + +Keep address-event listings bounded. `-n 0` and other unlimited settings materialize every matching row in memory, which can exhaust resources on a heavily reused address; start with `-n 100` and continue with the `offset` parameter, or narrow the window with `min_id`/`max_id` around the allocation event ID instead of requesting all rows at once. + +Interpret event actions as `4=alloc`, `5=free_requested`, and `6=free_completed`. Address reuse can produce multiple lifecycles, so correlate address, size, stream, allocation event ID, and event ordering before pairing events. + +When an unambiguous `free_requested` and `free_completed` pair exists, report their event IDs and event-ID distance. Event IDs are ordering markers, not timestamps; never report the distance as elapsed time. + +Do not use `block.state` as evidence for dynamic blocks. Its documented lifecycle meaning is reliable only for static blocks whose block ID is negative. + +### 6. Optionally establish a freed-block lifetime baseline + +The packaged templates do not aggregate `freeEventId - allocEventId` into lifetime buckets. Only when this baseline is needed, and only when the `sqlite3` CLI is available, use a read-only fallback after validating that `` is a non-negative decimal integer: + +```bash +sqlite3 -readonly '' " +SELECT CASE + WHEN freeEventId - allocEventId < 1000 THEN '<1k' + WHEN freeEventId - allocEventId < 5000 THEN '1k-5k' + WHEN freeEventId - allocEventId < 20000 THEN '5k-20k' + WHEN freeEventId - allocEventId < 100000 THEN '20k-100k' + ELSE '>=100k' + END AS lifetime_events, + COUNT(*) AS block_count, + SUM(size) AS size_bytes +FROM block_ +WHERE allocEventId >= 0 AND freeEventId >= allocEventId +GROUP BY lifetime_events +ORDER BY MIN(freeEventId - allocEventId);" +``` + +This baseline describes successfully freed blocks. It does not classify end-of-trace candidates and must not be generalized to real elapsed time. + +If `sqlite3` is unavailable or read-only mode cannot be verified, skip this optional step and list the lifetime distribution as unknown. Do not use a writable connection. + +### 7. Classify findings conservatively + +Use these result categories: + +- `strong leak candidate`: dynamic live bytes grow across comparable captures or phases, persist after expected cleanup, and have ownership evidence. Do not use this category from end-of-trace survival alone. +- `application retention`: a user-code allocation family stays referenced longer than expected, but eventual release has not been disproved. +- `framework lifecycle retention`: allocations appear tied to framework initialization, caches, workspaces, or finalization boundaries. +- `asynchronous free pending`: an unambiguous free request is present but completion is absent or occurs substantially later in event order. +- `allocator/cache effect`: reserved memory remains high while active memory does not show corresponding live allocation growth. +- `normal long-lived allocation`: lifetime matches the expected model, optimizer, graph, communication, or workspace lifecycle. +- `inconclusive`: capture duration, missing callstacks, static allocations, address reuse, or incomplete lifecycle evidence prevents classification. + +## Output Template + +Report results in this order: + +1. `Analysis scope`: database path, device ID, final event ID, and candidate threshold. +2. `Memory baseline`: separate allocated, active, and reserved peaks with their event IDs and same-event gaps. +3. `End-of-trace evidence`: dynamic candidate count, dynamic bytes by callstack, static bytes, and representative blocks. +4. `Peak-occupancy evidence`: callstack bytes and block counts at the active peak versus the final event, identity-matched blocks if any, and static plus `[preexisting live]` memory that carries no per-callstack attribution. +5. `Lifecycle evidence`: representative alloc, free-requested, and free-completed event IDs, with ambiguities called out. +6. `Findings`: category, confidence, evidence, inference, and affected callstack or allocation family. +7. `Unknowns`: missing timing, ownership, capture-boundary, callstack, or repeated-capture evidence. +8. `Suggested validation`: targeted experiments that could confirm or reject each leading hypothesis. + +Useful validation experiments include repeated snapshots at equivalent workload milestones, extending tracing beyond expected cleanup, explicitly releasing suspected application references, synchronizing the device before the capture ends, and comparing behavior before and after allocator cache cleanup. Explain that cache cleanup can change reserved memory without proving that application references were released. + +## Guardrails + +- Keep SnapshotDB access read-only. Never run `create`, `insert`, `update`, `delete`, `drop`, `alter`, `replace`, `attach`, `detach`, `reindex`, or `vacuum` SQL. +- Do not persist focus, configuration, reports, exports, scratch databases, or readiness files. +- Do not import or deserialize pickle snapshots. +- Prefer packaged `pt-snap` templates. Use raw SQLite only for the two documented read-only aggregates that templates do not expose: the freed-block lifetime baseline (Step 6) and the exact `[preexisting live]` total (Step 4). +- Do not call every allocation without a free event a leak. +- Separate cumulative allocation volume from memory simultaneously live at an event. +- Keep static memory separate from dynamic candidates. +- Do not treat event IDs as timestamps or event-ID distance as elapsed time. +- Do not generalize from one address, callstack, size class, or capture to all allocations. +- Keep result listings bounded; never request unlimited rows from a query. +- Treat peak-versus-final callstack aggregates as occupancy evidence only; claim persistence solely from identity-matched blocks. +- Label evidence, inference, and unknowns separately. + +## Verification Checklist + +- `pt-snap` availability was verified without installing or switching environments. +- Database and device were selected explicitly without changing focus. +- Final event and all three memory peaks were recorded. +- Same-event allocator gaps came from `allocator_gap`. +- End-of-trace dynamic candidates and static memory were reported separately. +- Active-peak and final-event callstack attribution used the same device. +- Peak-versus-final comparisons were reported as occupancy, and persistence claims relied on identity-matched blocks. +- Preexisting-live memory was reported separately from static and dynamic groups, with exact totals taken only from the documented read-only aggregates. +- Address-event listings stayed bounded with paging or event-ID windows. +- Database paths were single-quoted or passed through argument arrays without shell re-parsing. +- Representative lifecycle events were checked for address reuse and ambiguous pairing. +- Dynamic block state was not used as leak evidence. +- Event-ID distances were not presented as time durations. +- Every finding has a conservative category and confidence level. +- Unknowns and validation experiments are included. diff --git a/tests/AGENTS.md b/tests/AGENTS.md index 3cce773..c356fa9 100644 --- a/tests/AGENTS.md +++ b/tests/AGENTS.md @@ -20,6 +20,7 @@ | `test_fixture_provenance.py` | Non-deserializing coverage, SHA-256, size, and Git LFS pointer validation for executable fixtures. | | `test_governance.py` | Snapshot provenance and repository governance contracts. | | `test_mcp_server.py` | MCP server tool/resource/prompt behavior tests. | +| `test_memory_leak_skill.py` | Verifies the memory-leak skill's current template references, conservative classifications, and read-only boundaries. | | `test_memory_fragmentation_skill.py` | Verifies the fragmentation skill's current command/template references, runtime segment coverage, conservative classifications, and read-only boundaries. | | `test_models.py` | Package-level model behavior tests. | | `test_package.py` | Package metadata/import/version tests. | diff --git a/tests/test_memory_leak_skill.py b/tests/test_memory_leak_skill.py new file mode 100644 index 0000000..218d586 --- /dev/null +++ b/tests/test_memory_leak_skill.py @@ -0,0 +1,133 @@ +from pathlib import Path + +SKILL_PATH = Path("skills/pt-snap-memory-leak/SKILL.md") + + +def test_memory_leak_skill_uses_current_pt_snap_surfaces() -> None: + skill = SKILL_PATH.read_text() + + assert "name: pt-snap-memory-leak" in skill + assert "`pt-snap-setup`" in skill + assert "readiness.json" not in skill + + for template in ( + "memory_peak", + "allocator_gap", + "event", + "block", + "leak_detection", + "active_memory_callstack_at_event", + ): + assert f"--template-use {template}" in skill or f"--template-info {template}" in skill + + +def test_memory_leak_skill_preserves_diagnostic_boundaries() -> None: + skill = SKILL_PATH.read_text() + + required_guardrails = ( + "A single snapshot cannot confirm a leak.", + "Do not run `pt-snap import`", + "Do not run `pt-snap focus `", + "Do not use `block.state` as evidence for dynamic blocks.", + "Event IDs are ordering markers, not timestamps", + "Keep SnapshotDB access read-only.", + "Label evidence, inference, and unknowns separately.", + ) + for guardrail in required_guardrails: + assert guardrail in skill + + +def test_memory_leak_skill_uses_conservative_classifications() -> None: + skill = SKILL_PATH.read_text() + + for classification in ( + "strong leak candidate", + "application retention", + "framework lifecycle retention", + "asynchronous free pending", + "allocator/cache effect", + "normal long-lived allocation", + "inconclusive", + ): + assert f"`{classification}`" in skill + + assert "Do not use this category from end-of-trace survival alone." in skill + + +def test_memory_leak_skill_quotes_paths_safely() -> None: + skill = SKILL_PATH.read_text() + + assert "''" in skill + assert '""' not in skill + assert "Never place a substituted value inside double quotes" in skill + assert "argument array" in skill + assert "treat that value as untrusted input to quoting" in skill + + +def test_memory_leak_skill_has_no_unused_range_input() -> None: + skill = SKILL_PATH.read_text() + + assert "An event range to investigate" not in skill + assert "event range, final event ID" not in skill + + +def test_memory_leak_skill_reports_occupancy_with_identity_evidence() -> None: + skill = SKILL_PATH.read_text() + + assert "occupancy comparison" in skill + assert "not a block-identity survival test" in skill + assert "`top_n` truncation applies to dynamic callstack groups only" in skill + assert "static and preexisting groups are always returned in full" in skill + assert "Match representative blocks by identity" in skill + + +def test_memory_leak_skill_matches_current_preexisting_group_semantics() -> None: + skill = SKILL_PATH.read_text() + + assert "`[preexisting live] allocEventId=-1`" in skill + assert "[static] allocEventId=-1, freeEventId=-1" in skill + assert "are excluded from both groups" not in skill + assert "counts only blocks with `allocEventId != -1`" not in skill + + +def test_memory_leak_skill_cross_checks_preexisting_bucket_exactly() -> None: + skill = SKILL_PATH.read_text() + + assert "cannot express the `freeEventId IS NULL` case" in skill + assert "non-negative decimal integer matching `^[0-9]+$`" in skill + assert "COALESCE(SUM(size), 0) AS size_bytes" in skill + assert "freeEventId IS NULL" in skill + assert "OR freeEventId > " in skill + assert "(freeEventId < 0 AND freeEventId <> -1)" in skill + assert "returns a single row" in skill + assert "nothing is silently undercounted" in skill + + guardrail = "two documented read-only aggregates that templates do not expose" + assert guardrail in skill + assert "only for the optional lifetime aggregate" not in skill + assert "the freed-block lifetime baseline (Step 6)" in skill + assert "the exact `[preexisting live]` total (Step 4)" in skill + + +def test_memory_leak_skill_has_no_incomplete_bucket_scan() -> None: + skill = SKILL_PATH.read_text() + + assert '"allocEventId":-1,"min_freeEventId"' not in skill + assert "fall back to the paginated sum above" not in skill + + +def test_memory_leak_skill_keeps_result_listings_bounded() -> None: + skill = SKILL_PATH.read_text() + + address_query = '"order_dir":"ASC"}\' -n 100' + assert address_query in skill + assert "-n 0` and other unlimited settings materialize" in skill + assert "the `offset` parameter" in skill + assert "never request unlimited rows from a query" in skill + + +def test_memory_leak_skill_interprets_percent_column_as_byte_share() -> None: + skill = SKILL_PATH.read_text() + + assert "share of included active bytes" in skill + assert "byte share (`size_bytes / total size_bytes`) despite its name" in skill