diff --git a/skills/pt-snap-helper/SKILL.md b/skills/pt-snap-helper/SKILL.md index 865d4a6..bb9f5cc 100644 --- a/skills/pt-snap-helper/SKILL.md +++ b/skills/pt-snap-helper/SKILL.md @@ -94,10 +94,42 @@ value; `--opt=value` and numeric tokens do not consume the next argument). Text mode still writes `Error:` lines to stdout. This skill itself does not run analysis queries. The diagnostic skills own -those commands. For an existing SnapshotDB, point the next skill at -`pt-snap capabilities --json` and `pt-snap overview '' --json` -before diagnosis so it does not need per-template `--template-info` or a -separate metadata probe. +those commands and the shared preflight below, including when invoked directly. + +### Shared diagnostic preflight contract + +For an existing SnapshotDB, the normal prerequisite probes are +`pt-snap capabilities --json` and `pt-snap overview '' --json`. +The complete `templates[]` catalog supplies the same parameters, `output_schema`, +and field semantics as per-template `--template-info`; check every template +required by the selected skill by name. Stop on a missing required template or +failed probe and report the exact failure; do not substitute raw SQL. + +Read `overview.import_metadata.status` (the `import_metadata` field in the +overview JSON), not just exit code 0 or outer `ok: true`. Stop on `invalid` +metadata or schema errors. Continue with `available`, or with legacy +`unavailable` and reason `metadata_missing` after recording unknown import +provenance. Stop on any other status/reason or incomplete result. Overview also +provides devices and trace bounds; validate the selected device against it. + +Reuse complete successful results already obtained in this conversation only +for the same CLI/Python environment and unchanged database target. Capabilities +must match the current CLI version; overview must match the resolved absolute +database path. A summary, truncated output, changed environment/version, changed +database, or unknown result identity is not reusable. Fetch only the missing or +stale result. Direct invocation of a diagnostic skill performs the same complete +validation; helper use is not a prerequisite. Do not re-emit a reused catalog or +repeat per-template probes. Normal discovery costs 2 calls from cold, 1 with one +reusable result, and 0 with both; measure tool-call count and output bytes +separately, since the full catalog can be larger than individual probes. + +Capabilities does not describe report command options. Keep +`pt-snap report peak-memory --help` when the selected workflow requires report +validation (mandatory for peak breakdown, conditional for fragmentation). +Availability/focus/report-help checks are separate from the two discovery calls. +Missing-focus recovery below is the explicit metadata-first exception: after +user confirmation, inspect metadata first, then obtain/reuse capabilities and +overview and apply these same checks. Metadata alone does not replace overview. ## Routing matrix @@ -131,7 +163,8 @@ A SnapshotDB is a pt-snap SQLite database (typically `.db`) with a 1. Confirm the goal is leak, peak, or fragmentation. 2. Tell the next skill to start with `pt-snap capabilities --json` and `pt-snap overview '' --json` (overview first, then diagnose). - Pass the database path and device if the user supplied them. + Pass the database path and device if the user supplied them, plus any complete + reusable results under the shared preflight contract above. 3. Hand off to the matching diagnostic skill. Do not run leak, peak, or fragmentation query templates from this helper. 4. Do not run `pt-snap focus ` to persist a new focus from this @@ -166,6 +199,12 @@ explicit path and do not enter this branch. re-resolve device from the confirmed database. This helper must not run metadata, query, import, or persist focus. +The diagnostic skill must stop on invalid metadata or schema errors in this +metadata-first exception. Legacy `unavailable` with reason `metadata_missing` +permits continuing with unknown import provenance recorded. Then apply the +shared capabilities/overview preflight; do not reuse the missing target's +overview or treat metadata as an overview substitute. + This helper may mention that `pt-snap overview '' --json` and `pt-snap capabilities --json` exist. Running those orientation commands belongs to the diagnostic skill. Do not import pickle or persist focus. diff --git a/skills/pt-snap-memory-fragmentation/SKILL.md b/skills/pt-snap-memory-fragmentation/SKILL.md index abea016..0edad93 100644 --- a/skills/pt-snap-memory-fragmentation/SKILL.md +++ b/skills/pt-snap-memory-fragmentation/SKILL.md @@ -99,30 +99,58 @@ explicit path and do not enter this branch. reason `metadata_missing`; record unknown import provenance and do not force a re-import. Stop on schema errors or invalid metadata. -### 3. Verify metadata and required templates before diagnosis +### 3. Verify capabilities and the database in two calls -Run these checks before any diagnostic query: +Apply the shared diagnostic preflight contract in `pt-snap-helper`, even on +direct invocation. Run only the probes whose complete results are not reusable: ```bash -pt-snap metadata "" --json -pt-snap query --template-info memory_peak -pt-snap query --template-info allocator_gap -pt-snap query --template-info allocation -pt-snap query --template-info event +pt-snap capabilities --json +pt-snap overview "" --json ``` -Stop on invalid database metadata/schema or a missing required template, and -report the exact failure. A compatible legacy database may report metadata as -unavailable; record that provenance limitation rather than treating it as -import metadata. Do not replace a missing required template with raw SQL. - -Before using the optional attribution phase, also verify its surfaces: +Reuse complete successful results from this conversation only for the same +CLI/Python environment and unchanged database target. Check CLI version and the +overview's resolved absolute database path. Summaries, truncated output, changed +environment/version or database, and unknown result identity require refreshing +the affected probe. Revalidate the contents even when reusing results. + +Read `overview.import_metadata.status` (the `import_metadata` field in the +overview JSON), not just exit code 0 or outer `ok: true`. Stop on `invalid` +metadata or schema errors. Continue with `available`, or with legacy +`unavailable` and reason `metadata_missing` after recording unknown import +provenance. Stop on any other status/reason or incomplete result. Validate the +selected device against overview's devices and record its trace bounds. + +Confirm these templates exist in the capabilities catalog before diagnosis: +`memory_peak`, `allocator_gap`, `allocation`, `event`. Read parameters, +`output_schema`, and field semantics from those entries. Stop on a missing +required template or failed probe and report the exact failure. +Do not replace a missing required template with raw SQL. Do not probe templates +one-by-one with `--template-info` or re-emit a reused catalog. + +Prerequisite probe budget: 2 calls from cold, 1 with one reusable result, 0 with +both. Record tool-call count and output bytes separately; fewer calls need not +mean fewer bytes. Availability/focus/report-help checks are outside this budget. +Missing-focus recovery is the metadata-first exception: after user confirmation, +run metadata and apply its stop/legacy rules above, then obtain/reuse capabilities +and overview. Metadata alone does not replace overview; never reuse the missing +target's overview or device. + +Before using the optional attribution phase, also verify +`active_memory_callstack_at_event` in the retained capabilities catalog; stop +that phase and report the missing template if absent. If using the report, +verify its command options with: ```bash -pt-snap query --template-info active_memory_callstack_at_event pt-snap report peak-memory --help ``` +Capabilities does not describe report command options, so keep this help check. +Stop the report phase and report the exact failure if it fails or the options +used below are absent. A complete successful help result from the same +CLI/Python environment and version may be reused. + ## Diagnostic Workflow ### 1. Establish separate peaks and same-event gaps diff --git a/skills/pt-snap-memory-leak/SKILL.md b/skills/pt-snap-memory-leak/SKILL.md index 9af2fd9..f90152e 100644 --- a/skills/pt-snap-memory-leak/SKILL.md +++ b/skills/pt-snap-memory-leak/SKILL.md @@ -67,18 +67,36 @@ Enter this recovery branch only when diagnosis depends on the current effective ### 3. Verify capabilities and the database in two calls -Run: +Apply the shared diagnostic preflight contract in `pt-snap-helper`, even on +direct invocation. Run only the probes whose complete results are not reusable: ```bash pt-snap capabilities --json pt-snap overview '' --json ``` -`capabilities --json` is the catalog: CLI version, every template contract (parameters, `output_schema`, field semantics), and the skill list. Do not probe templates one-by-one with `--template-info`. - -`overview --json` is the read-only database orientation: device list, per-device first/last event id, and import-metadata status. Do not persist focus. - -Prerequisite probe budget: previously 7 calls (`metadata` plus six `--template-info` probes). Now 2 calls (`capabilities` + `overview`). Record that reduction when evaluating this skill. +Reuse complete successful results from this conversation only for the same +CLI/Python environment and unchanged database target. Check CLI version and the +overview's resolved absolute database path. Summaries, truncated output, changed +environment/version or database, and unknown result identity require refreshing +the affected probe. Revalidate the contents even when reusing results. + +`capabilities --json` is the catalog: CLI version, every template contract (parameters, `output_schema`, field semantics), and the skill list. Do not probe templates one-by-one with `--template-info`. Do not re-emit a reused catalog. + +Read `overview.import_metadata.status` (the `import_metadata` field in the +overview JSON), not just exit code 0 or outer `ok: true`. Stop on `invalid` +metadata or schema errors. Continue with `available`, or with legacy +`unavailable` and reason `metadata_missing` after recording unknown import +provenance. Stop on any other status/reason or incomplete result. Validate the +selected device against overview's devices and record its trace bounds. + +Prerequisite probe budget: 2 calls from cold, 1 with one reusable result, 0 with +both. Record tool-call count and output bytes separately; fewer calls need not +mean fewer bytes. Availability/focus checks are outside this discovery budget. +Missing-focus recovery is the metadata-first exception: after user confirmation, +run metadata and apply its stop/legacy rules above, then obtain/reuse capabilities +and overview. Metadata alone does not replace overview; never reuse the missing +target's overview or device. Confirm these templates exist in the capabilities catalog before diagnosis: `memory_peak`, `allocator_gap`, `event`, `block`, `leak_detection`, `active_memory_callstack_at_event`, `preexisting_live`, `freed_block_lifetime`. If the catalog or overview fails, stop and report the exact failure. Do not silently substitute raw SQL for a missing core template. diff --git a/skills/pt-snap-memory-peak-breakdown/SKILL.md b/skills/pt-snap-memory-peak-breakdown/SKILL.md index 9d2ad8e..d53bf2e 100644 --- a/skills/pt-snap-memory-peak-breakdown/SKILL.md +++ b/skills/pt-snap-memory-peak-breakdown/SKILL.md @@ -108,36 +108,56 @@ explicit path and do not enter this branch. reason `metadata_missing`; record unknown import provenance and do not force a re-import. Stop on schema errors or invalid metadata. -### 3. Verify SnapshotDB metadata +### 3. Verify capabilities and the database in two calls -Run with the resolved absolute database path: +Apply the shared diagnostic preflight contract in `pt-snap-helper`, even on +direct invocation. Run only the probes whose complete results are not reusable: ```bash -pt-snap metadata "" --json +pt-snap capabilities --json +pt-snap overview "" --json ``` -Stop if the command rejects the database schema or reports invalid metadata. A -status of `unavailable` can represent a schema-valid legacy SnapshotDB; proceed -only after recording import provenance as unknown. Do not search for a source -pickle. - -### 4. Verify the report command and required templates +Reuse complete successful results from this conversation only for the same +CLI/Python environment and unchanged database target. Check CLI version and the +overview's resolved absolute database path. Summaries, truncated output, changed +environment/version or database, and unknown result identity require refreshing +the affected probe. Revalidate the contents even when reusing results. + +Read `overview.import_metadata.status` (the `import_metadata` field in the +overview JSON), not just exit code 0 or outer `ok: true`. Stop on `invalid` +metadata or schema errors. Continue with `available`, or with legacy +`unavailable` and reason `metadata_missing` after recording unknown import +provenance. Stop on any other status/reason or incomplete result. Validate the +selected device against overview's devices and record its trace bounds. + +Confirm these templates exist in the capabilities catalog before diagnosis: +`memory_peak`, `allocator_gap`, `active_memory_callstack_at_event`, +`active_blocks_at_event`. Read parameters, `output_schema`, and field semantics +from those entries. Stop on a missing required template or failed probe and +report the exact failure. Do not substitute raw SQL. Do not probe templates +one-by-one with `--template-info` or re-emit a reused catalog. + +Prerequisite probe budget: 2 calls from cold, 1 with one reusable result, 0 with +both. Record tool-call count and output bytes separately; fewer calls need not +mean fewer bytes. Availability/focus/report-help checks are outside this budget. +Missing-focus recovery is the metadata-first exception: after user confirmation, +run metadata and apply its stop/legacy rules above, then obtain/reuse capabilities +and overview. Metadata alone does not replace overview; never reuse the missing +target's overview or device. + +### 4. Verify the report command Run: ```bash pt-snap report peak-memory --help -pt-snap query --template-info memory_peak -pt-snap query --template-info allocator_gap -pt-snap query --template-info active_memory_callstack_at_event -pt-snap query --template-info active_blocks_at_event ``` -Stop and report the missing command or template if any check fails. A missing -template prints an error but still exits with status 0, so never rely on exit -codes for these checks: each template check succeeds only when its output -contains the expected `Template: ` record. Do not replace the -productized workflow with ad hoc SQL. +Capabilities does not describe report command options, so keep this help check. +Stop and report the exact failure if it fails or the options used below are +absent. A complete successful help result from the same CLI/Python environment +and version may be reused. ## Full-Trace Workflow @@ -229,7 +249,7 @@ their event ID carries the range selection forward. - It does not assign reserved/cache bytes, allocator gaps, or inactive allocated bytes to callstacks. A large gap is evidence of counter separation at one event, not proof of fragmentation, caching policy, or an OOM cause. -- Interpret `percent_of_active_blocks` using `--template-info active_memory_callstack_at_event` (units, denominator, and limits). Do not infer a block-count share from the column name. State the `include_static` choice whenever percentages are reported. +- Interpret `percent_of_active_blocks` using the `active_memory_callstack_at_event` entry in the retained capabilities catalog (units, denominator, and limits). Do not infer a block-count share from the column name. State the `include_static` choice whenever percentages are reported. ### Static, preexisting, and dynamic memory diff --git a/src/pt_snap_cli/bundled_skills/pt-snap-helper/SKILL.md b/src/pt_snap_cli/bundled_skills/pt-snap-helper/SKILL.md index 865d4a6..bb9f5cc 100644 --- a/src/pt_snap_cli/bundled_skills/pt-snap-helper/SKILL.md +++ b/src/pt_snap_cli/bundled_skills/pt-snap-helper/SKILL.md @@ -94,10 +94,42 @@ value; `--opt=value` and numeric tokens do not consume the next argument). Text mode still writes `Error:` lines to stdout. This skill itself does not run analysis queries. The diagnostic skills own -those commands. For an existing SnapshotDB, point the next skill at -`pt-snap capabilities --json` and `pt-snap overview '' --json` -before diagnosis so it does not need per-template `--template-info` or a -separate metadata probe. +those commands and the shared preflight below, including when invoked directly. + +### Shared diagnostic preflight contract + +For an existing SnapshotDB, the normal prerequisite probes are +`pt-snap capabilities --json` and `pt-snap overview '' --json`. +The complete `templates[]` catalog supplies the same parameters, `output_schema`, +and field semantics as per-template `--template-info`; check every template +required by the selected skill by name. Stop on a missing required template or +failed probe and report the exact failure; do not substitute raw SQL. + +Read `overview.import_metadata.status` (the `import_metadata` field in the +overview JSON), not just exit code 0 or outer `ok: true`. Stop on `invalid` +metadata or schema errors. Continue with `available`, or with legacy +`unavailable` and reason `metadata_missing` after recording unknown import +provenance. Stop on any other status/reason or incomplete result. Overview also +provides devices and trace bounds; validate the selected device against it. + +Reuse complete successful results already obtained in this conversation only +for the same CLI/Python environment and unchanged database target. Capabilities +must match the current CLI version; overview must match the resolved absolute +database path. A summary, truncated output, changed environment/version, changed +database, or unknown result identity is not reusable. Fetch only the missing or +stale result. Direct invocation of a diagnostic skill performs the same complete +validation; helper use is not a prerequisite. Do not re-emit a reused catalog or +repeat per-template probes. Normal discovery costs 2 calls from cold, 1 with one +reusable result, and 0 with both; measure tool-call count and output bytes +separately, since the full catalog can be larger than individual probes. + +Capabilities does not describe report command options. Keep +`pt-snap report peak-memory --help` when the selected workflow requires report +validation (mandatory for peak breakdown, conditional for fragmentation). +Availability/focus/report-help checks are separate from the two discovery calls. +Missing-focus recovery below is the explicit metadata-first exception: after +user confirmation, inspect metadata first, then obtain/reuse capabilities and +overview and apply these same checks. Metadata alone does not replace overview. ## Routing matrix @@ -131,7 +163,8 @@ A SnapshotDB is a pt-snap SQLite database (typically `.db`) with a 1. Confirm the goal is leak, peak, or fragmentation. 2. Tell the next skill to start with `pt-snap capabilities --json` and `pt-snap overview '' --json` (overview first, then diagnose). - Pass the database path and device if the user supplied them. + Pass the database path and device if the user supplied them, plus any complete + reusable results under the shared preflight contract above. 3. Hand off to the matching diagnostic skill. Do not run leak, peak, or fragmentation query templates from this helper. 4. Do not run `pt-snap focus ` to persist a new focus from this @@ -166,6 +199,12 @@ explicit path and do not enter this branch. re-resolve device from the confirmed database. This helper must not run metadata, query, import, or persist focus. +The diagnostic skill must stop on invalid metadata or schema errors in this +metadata-first exception. Legacy `unavailable` with reason `metadata_missing` +permits continuing with unknown import provenance recorded. Then apply the +shared capabilities/overview preflight; do not reuse the missing target's +overview or treat metadata as an overview substitute. + This helper may mention that `pt-snap overview '' --json` and `pt-snap capabilities --json` exist. Running those orientation commands belongs to the diagnostic skill. Do not import pickle or persist focus. diff --git a/src/pt_snap_cli/bundled_skills/pt-snap-memory-fragmentation/SKILL.md b/src/pt_snap_cli/bundled_skills/pt-snap-memory-fragmentation/SKILL.md index abea016..0edad93 100644 --- a/src/pt_snap_cli/bundled_skills/pt-snap-memory-fragmentation/SKILL.md +++ b/src/pt_snap_cli/bundled_skills/pt-snap-memory-fragmentation/SKILL.md @@ -99,30 +99,58 @@ explicit path and do not enter this branch. reason `metadata_missing`; record unknown import provenance and do not force a re-import. Stop on schema errors or invalid metadata. -### 3. Verify metadata and required templates before diagnosis +### 3. Verify capabilities and the database in two calls -Run these checks before any diagnostic query: +Apply the shared diagnostic preflight contract in `pt-snap-helper`, even on +direct invocation. Run only the probes whose complete results are not reusable: ```bash -pt-snap metadata "" --json -pt-snap query --template-info memory_peak -pt-snap query --template-info allocator_gap -pt-snap query --template-info allocation -pt-snap query --template-info event +pt-snap capabilities --json +pt-snap overview "" --json ``` -Stop on invalid database metadata/schema or a missing required template, and -report the exact failure. A compatible legacy database may report metadata as -unavailable; record that provenance limitation rather than treating it as -import metadata. Do not replace a missing required template with raw SQL. - -Before using the optional attribution phase, also verify its surfaces: +Reuse complete successful results from this conversation only for the same +CLI/Python environment and unchanged database target. Check CLI version and the +overview's resolved absolute database path. Summaries, truncated output, changed +environment/version or database, and unknown result identity require refreshing +the affected probe. Revalidate the contents even when reusing results. + +Read `overview.import_metadata.status` (the `import_metadata` field in the +overview JSON), not just exit code 0 or outer `ok: true`. Stop on `invalid` +metadata or schema errors. Continue with `available`, or with legacy +`unavailable` and reason `metadata_missing` after recording unknown import +provenance. Stop on any other status/reason or incomplete result. Validate the +selected device against overview's devices and record its trace bounds. + +Confirm these templates exist in the capabilities catalog before diagnosis: +`memory_peak`, `allocator_gap`, `allocation`, `event`. Read parameters, +`output_schema`, and field semantics from those entries. Stop on a missing +required template or failed probe and report the exact failure. +Do not replace a missing required template with raw SQL. Do not probe templates +one-by-one with `--template-info` or re-emit a reused catalog. + +Prerequisite probe budget: 2 calls from cold, 1 with one reusable result, 0 with +both. Record tool-call count and output bytes separately; fewer calls need not +mean fewer bytes. Availability/focus/report-help checks are outside this budget. +Missing-focus recovery is the metadata-first exception: after user confirmation, +run metadata and apply its stop/legacy rules above, then obtain/reuse capabilities +and overview. Metadata alone does not replace overview; never reuse the missing +target's overview or device. + +Before using the optional attribution phase, also verify +`active_memory_callstack_at_event` in the retained capabilities catalog; stop +that phase and report the missing template if absent. If using the report, +verify its command options with: ```bash -pt-snap query --template-info active_memory_callstack_at_event pt-snap report peak-memory --help ``` +Capabilities does not describe report command options, so keep this help check. +Stop the report phase and report the exact failure if it fails or the options +used below are absent. A complete successful help result from the same +CLI/Python environment and version may be reused. + ## Diagnostic Workflow ### 1. Establish separate peaks and same-event gaps diff --git a/src/pt_snap_cli/bundled_skills/pt-snap-memory-leak/SKILL.md b/src/pt_snap_cli/bundled_skills/pt-snap-memory-leak/SKILL.md index 9af2fd9..f90152e 100644 --- a/src/pt_snap_cli/bundled_skills/pt-snap-memory-leak/SKILL.md +++ b/src/pt_snap_cli/bundled_skills/pt-snap-memory-leak/SKILL.md @@ -67,18 +67,36 @@ Enter this recovery branch only when diagnosis depends on the current effective ### 3. Verify capabilities and the database in two calls -Run: +Apply the shared diagnostic preflight contract in `pt-snap-helper`, even on +direct invocation. Run only the probes whose complete results are not reusable: ```bash pt-snap capabilities --json pt-snap overview '' --json ``` -`capabilities --json` is the catalog: CLI version, every template contract (parameters, `output_schema`, field semantics), and the skill list. Do not probe templates one-by-one with `--template-info`. - -`overview --json` is the read-only database orientation: device list, per-device first/last event id, and import-metadata status. Do not persist focus. - -Prerequisite probe budget: previously 7 calls (`metadata` plus six `--template-info` probes). Now 2 calls (`capabilities` + `overview`). Record that reduction when evaluating this skill. +Reuse complete successful results from this conversation only for the same +CLI/Python environment and unchanged database target. Check CLI version and the +overview's resolved absolute database path. Summaries, truncated output, changed +environment/version or database, and unknown result identity require refreshing +the affected probe. Revalidate the contents even when reusing results. + +`capabilities --json` is the catalog: CLI version, every template contract (parameters, `output_schema`, field semantics), and the skill list. Do not probe templates one-by-one with `--template-info`. Do not re-emit a reused catalog. + +Read `overview.import_metadata.status` (the `import_metadata` field in the +overview JSON), not just exit code 0 or outer `ok: true`. Stop on `invalid` +metadata or schema errors. Continue with `available`, or with legacy +`unavailable` and reason `metadata_missing` after recording unknown import +provenance. Stop on any other status/reason or incomplete result. Validate the +selected device against overview's devices and record its trace bounds. + +Prerequisite probe budget: 2 calls from cold, 1 with one reusable result, 0 with +both. Record tool-call count and output bytes separately; fewer calls need not +mean fewer bytes. Availability/focus checks are outside this discovery budget. +Missing-focus recovery is the metadata-first exception: after user confirmation, +run metadata and apply its stop/legacy rules above, then obtain/reuse capabilities +and overview. Metadata alone does not replace overview; never reuse the missing +target's overview or device. Confirm these templates exist in the capabilities catalog before diagnosis: `memory_peak`, `allocator_gap`, `event`, `block`, `leak_detection`, `active_memory_callstack_at_event`, `preexisting_live`, `freed_block_lifetime`. If the catalog or overview fails, stop and report the exact failure. Do not silently substitute raw SQL for a missing core template. diff --git a/src/pt_snap_cli/bundled_skills/pt-snap-memory-peak-breakdown/SKILL.md b/src/pt_snap_cli/bundled_skills/pt-snap-memory-peak-breakdown/SKILL.md index 9d2ad8e..d53bf2e 100644 --- a/src/pt_snap_cli/bundled_skills/pt-snap-memory-peak-breakdown/SKILL.md +++ b/src/pt_snap_cli/bundled_skills/pt-snap-memory-peak-breakdown/SKILL.md @@ -108,36 +108,56 @@ explicit path and do not enter this branch. reason `metadata_missing`; record unknown import provenance and do not force a re-import. Stop on schema errors or invalid metadata. -### 3. Verify SnapshotDB metadata +### 3. Verify capabilities and the database in two calls -Run with the resolved absolute database path: +Apply the shared diagnostic preflight contract in `pt-snap-helper`, even on +direct invocation. Run only the probes whose complete results are not reusable: ```bash -pt-snap metadata "" --json +pt-snap capabilities --json +pt-snap overview "" --json ``` -Stop if the command rejects the database schema or reports invalid metadata. A -status of `unavailable` can represent a schema-valid legacy SnapshotDB; proceed -only after recording import provenance as unknown. Do not search for a source -pickle. - -### 4. Verify the report command and required templates +Reuse complete successful results from this conversation only for the same +CLI/Python environment and unchanged database target. Check CLI version and the +overview's resolved absolute database path. Summaries, truncated output, changed +environment/version or database, and unknown result identity require refreshing +the affected probe. Revalidate the contents even when reusing results. + +Read `overview.import_metadata.status` (the `import_metadata` field in the +overview JSON), not just exit code 0 or outer `ok: true`. Stop on `invalid` +metadata or schema errors. Continue with `available`, or with legacy +`unavailable` and reason `metadata_missing` after recording unknown import +provenance. Stop on any other status/reason or incomplete result. Validate the +selected device against overview's devices and record its trace bounds. + +Confirm these templates exist in the capabilities catalog before diagnosis: +`memory_peak`, `allocator_gap`, `active_memory_callstack_at_event`, +`active_blocks_at_event`. Read parameters, `output_schema`, and field semantics +from those entries. Stop on a missing required template or failed probe and +report the exact failure. Do not substitute raw SQL. Do not probe templates +one-by-one with `--template-info` or re-emit a reused catalog. + +Prerequisite probe budget: 2 calls from cold, 1 with one reusable result, 0 with +both. Record tool-call count and output bytes separately; fewer calls need not +mean fewer bytes. Availability/focus/report-help checks are outside this budget. +Missing-focus recovery is the metadata-first exception: after user confirmation, +run metadata and apply its stop/legacy rules above, then obtain/reuse capabilities +and overview. Metadata alone does not replace overview; never reuse the missing +target's overview or device. + +### 4. Verify the report command Run: ```bash pt-snap report peak-memory --help -pt-snap query --template-info memory_peak -pt-snap query --template-info allocator_gap -pt-snap query --template-info active_memory_callstack_at_event -pt-snap query --template-info active_blocks_at_event ``` -Stop and report the missing command or template if any check fails. A missing -template prints an error but still exits with status 0, so never rely on exit -codes for these checks: each template check succeeds only when its output -contains the expected `Template: ` record. Do not replace the -productized workflow with ad hoc SQL. +Capabilities does not describe report command options, so keep this help check. +Stop and report the exact failure if it fails or the options used below are +absent. A complete successful help result from the same CLI/Python environment +and version may be reused. ## Full-Trace Workflow @@ -229,7 +249,7 @@ their event ID carries the range selection forward. - It does not assign reserved/cache bytes, allocator gaps, or inactive allocated bytes to callstacks. A large gap is evidence of counter separation at one event, not proof of fragmentation, caching policy, or an OOM cause. -- Interpret `percent_of_active_blocks` using `--template-info active_memory_callstack_at_event` (units, denominator, and limits). Do not infer a block-count share from the column name. State the `include_static` choice whenever percentages are reported. +- Interpret `percent_of_active_blocks` using the `active_memory_callstack_at_event` entry in the retained capabilities catalog (units, denominator, and limits). Do not infer a block-count share from the column name. State the `include_static` choice whenever percentages are reported. ### Static, preexisting, and dynamic memory diff --git a/tests/skills/test_memory_fragmentation_contract.py b/tests/skills/test_memory_fragmentation_contract.py index f976a56..8f77052 100644 --- a/tests/skills/test_memory_fragmentation_contract.py +++ b/tests/skills/test_memory_fragmentation_contract.py @@ -31,7 +31,7 @@ def test_memory_fragmentation_skill_uses_current_pt_snap_surfaces() -> None: for template, path in templates.items(): assert path.is_file() assert f" {template}:" in path.read_text() - assert f"--template-info {template}" in skill + assert f"`{template}`" in skill.split("## Diagnostic Workflow")[0] assert f"--template-use {template}" in skill diff --git a/tests/skills/test_memory_leak_contract.py b/tests/skills/test_memory_leak_contract.py index ae0dbf1..3ada303 100644 --- a/tests/skills/test_memory_leak_contract.py +++ b/tests/skills/test_memory_leak_contract.py @@ -140,10 +140,10 @@ def test_memory_leak_skill_interprets_percent_column_as_byte_share() -> None: def test_memory_leak_skill_records_reduced_prerequisite_probes() -> None: - skill = SKILL_PATH.read_text() + skill = " ".join(SKILL_PATH.read_text().split()) - assert "previously 7 calls" in skill - assert "Now 2 calls" in skill + assert "2 calls from cold, 1 with one reusable result, 0 with both" in skill + assert "Record tool-call count and output bytes separately" in skill assert "Do not probe templates one-by-one with `--template-info`" in skill diff --git a/tests/skills/test_memory_peak_breakdown_contract.py b/tests/skills/test_memory_peak_breakdown_contract.py index 4eb0113..b6b7380 100644 --- a/tests/skills/test_memory_peak_breakdown_contract.py +++ b/tests/skills/test_memory_peak_breakdown_contract.py @@ -41,7 +41,7 @@ def test_skill_references_current_report_and_templates() -> None: assert "pt-snap report peak-memory --help" in skill for name, path in templates.items(): assert f" {name}:" in path.read_text(encoding="utf-8") - assert f"--template-info {name}" in skill + assert f"`{name}`" in skill.split("## Full-Trace Workflow")[0] assert "has_more" in skill assert "-n " in skill assert "increase `top_n`" in skill @@ -102,12 +102,15 @@ def test_placeholder_values_are_validated_before_shell_substitution() -> None: assert "Prefer argument-array execution" in normalized -def test_preflight_checks_template_records_not_exit_codes() -> None: +def test_preflight_checks_catalog_entries_and_keeps_report_help() -> None: normalized = _normalized_skill() - assert "still exits with status 0" in normalized - assert "never rely on exit codes for these checks" in normalized - assert "expected `Template: ` record" in normalized + assert ( + "Confirm these templates exist in the capabilities catalog before diagnosis" in normalized + ) + assert "Stop on a missing required template or failed probe" in normalized + assert "Capabilities does not describe report command options" in normalized + assert "pt-snap report peak-memory --help" in normalized def test_negative_event_ids_are_initial_state_reconstruction() -> None: @@ -173,7 +176,7 @@ def test_attribution_caveats_prevent_reserved_and_static_overclaim() -> None: assert "when the selected event is the allocated or reserved peak" in normalized assert "It does not assign reserved/cache bytes" in skill assert ( - "Interpret `percent_of_active_blocks` using `--template-info active_memory_callstack_at_event`" + "Interpret `percent_of_active_blocks` using the `active_memory_callstack_at_event` entry in the retained capabilities catalog" in skill ) assert "Do not infer a block-count share from the column name" in skill diff --git a/tests/skills/test_preflight_contract.py b/tests/skills/test_preflight_contract.py new file mode 100644 index 0000000..794915e --- /dev/null +++ b/tests/skills/test_preflight_contract.py @@ -0,0 +1,191 @@ +"""Static agent policy plus real CLI payloads; no live model execution.""" + +import json +import re +import shlex +import sqlite3 +from pathlib import Path + +import pytest +from typer.testing import CliRunner + +from pt_snap_cli.cli import app +from pt_snap_cli.core.import_metadata import ImportMetadataService +from pt_snap_cli.query import registry + +ROOT = Path(__file__).resolve().parents[2] +DIAGNOSTICS = ( + "pt-snap-memory-leak", + "pt-snap-memory-peak-breakdown", + "pt-snap-memory-fragmentation", +) +runner = CliRunner() + + +def _skill(name: str) -> str: + return (ROOT / "skills" / name / "SKILL.md").read_text() + + +def _preflight(name: str) -> str: + return _skill(name).split("### 3. Verify capabilities", 1)[1].split("\n## ", 1)[0] + + +def _required(name: str) -> set[str]: + sentence = ( + _preflight(name) + .split("Confirm these templates exist in the capabilities catalog before diagnosis:", 1)[1] + .split(".", 1)[0] + ) + return set(re.findall(r"`(\w+)`", sentence)) + + +@pytest.mark.parametrize("name", ("pt-snap-helper", *DIAGNOSTICS)) +def test_shared_preflight_checks_nested_metadata_and_reuse_identity(name: str) -> None: + text = " ".join(_skill(name).split()) + for instruction in ( + "`overview.import_metadata.status`", + "not just exit code 0 or outer `ok: true`", + "Stop on `invalid` metadata or schema errors", + "`unavailable` and reason `metadata_missing`", + "recording unknown import provenance", + "Stop on any other status/reason or incomplete result", + "CLI/Python environment and unchanged database target", + "CLI version", + "resolved absolute database path", + "truncated output", + "unknown result identity", + "metadata-first exception", + "Metadata alone does not replace overview", + "tool-call count and output bytes", + ): + assert instruction in text + + +@pytest.mark.parametrize("name", DIAGNOSTICS) +def test_direct_invocation_validates_even_reused_results_without_extra_probes(name: str) -> None: + text = " ".join(_preflight(name).split()) + assert "even on direct invocation" in text + assert "Run only the probes whose complete results are not reusable" in text + assert "Revalidate the contents even when reusing results" in text + assert "2 calls from cold, 1 with one reusable result, 0 with both" in text + assert "re-emit a reused catalog" in text + assert "never reuse the missing target's overview or device" in text + assert "missing" in text and "template" in text and "Stop" in text + commands = [ + shlex.split(line) + for block in re.findall(r"```bash\n(.*?)```", _preflight(name), re.S) + for line in block.strip().splitlines() + ] + discovery = [cmd for cmd in commands if cmd[1] != "report"] + assert discovery == [ + ["pt-snap", "capabilities", "--json"], + ["pt-snap", "overview", "" if "peak" in name else "", "--json"], + ] + if name != "pt-snap-memory-leak": + assert commands.count(["pt-snap", "report", "peak-memory", "--help"]) == 1 + assert "Capabilities does not describe report command options" in text + if name == "pt-snap-memory-fragmentation": + assert "If using the report" in text + assert "stop that phase and report the missing template if absent" in text + + +@pytest.fixture +def db(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path: + monkeypatch.chdir(tmp_path) + monkeypatch.setattr(Path, "home", lambda: tmp_path) + monkeypatch.delenv("PT_SNAP_DB_PATH", raising=False) + monkeypatch.delenv("PT_SNAP_SKILLS_DIR", raising=False) + path = tmp_path / "preflight.db" + with sqlite3.connect(path) as conn: + conn.execute( + "CREATE TABLE dictionary (`table` TEXT, `column` TEXT, `key` TEXT, `value` TEXT)" + ) + conn.execute("CREATE TABLE trace_entry_0 (id INTEGER PRIMARY KEY)") + conn.execute("INSERT INTO trace_entry_0 VALUES (7)") + return path + + +@pytest.mark.parametrize("name", DIAGNOSTICS) +def test_documented_discovery_commands_supply_contracts_and_measure_output( + name: str, db: Path, record_property +) -> None: + # Execute the actual fenced discovery block, not a separately invented plan. + block = re.findall(r"```bash\n(.*?)```", _preflight(name), re.S)[0] + commands = [ + shlex.split(line.replace("", str(db)).replace("", str(db)))[1:] + for line in block.strip().splitlines() + ] + results = [runner.invoke(app, args) for args in commands] + assert len(results) == 2 + assert all(result.exit_code == 0 and not result.stderr for result in results) + catalog, overview = [json.loads(result.stdout) for result in results] + entries = {entry["name"]: entry for entry in catalog["templates"]} + assert _required(name) <= entries.keys() + assert overview["db_path"] == str(db.resolve()) + assert overview["devices"] == [{"device_id": 0, "first_event_id": 7, "last_event_id": 7}] + + # Equivalence justifies omitting these calls in the agent workflow. Count + # their bytes separately: fewer calls does not imply a smaller cold catalog. + redundant = [runner.invoke(app, ["metadata", str(db), "--json"])] + for template in sorted(_required(name)): + result = runner.invoke(app, ["query", "--template-info", template, "--json"]) + assert result.exit_code == 0 + info = json.loads(result.stdout) + assert entries[template] == {key: info[key] for key in entries[template]} + redundant.append(result) + metadata = json.loads(redundant[0].stdout) + assert overview["import_metadata"]["status"] == metadata["status"] == "unavailable" + assert overview["import_metadata"]["reason"] == metadata["reason"] == "metadata_missing" + discovery_bytes = sum(len(result.stdout.encode()) for result in results) + redundant_bytes = sum(len(result.stdout.encode()) for result in redundant) + assert discovery_bytes > 0 and redundant_bytes > 0 + record_property("discovery_calls", len(results)) + record_property("discovery_output_bytes", discovery_bytes) + record_property("avoided_redundant_calls", len(redundant)) + record_property("avoided_redundant_output_bytes", redundant_bytes) + + +@pytest.mark.parametrize("name", DIAGNOSTICS) +def test_successful_catalog_can_be_missing_a_required_template( + name: str, db: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + # Simulate an installed catalog without a required core template. Outer + # success is not sufficient; the skill's named-entry check must reject it. + monkeypatch.delitem(registry._registry._queries, "memory_peak") + result = runner.invoke(app, ["capabilities", "--json"]) + assert result.exit_code == 0 + catalog = json.loads(result.stdout) + assert catalog["ok"] is True + names = {entry["name"] for entry in catalog["templates"]} + assert _required(name) - names == {"memory_peak"} + + +@pytest.mark.parametrize("state", ("legacy", "available", "invalid", "schema-error")) +def test_overview_status_drives_preflight_even_with_outer_success(db: Path, state: str) -> None: + if state == "available": + # Metadata construction does not deserialize any input. + source = db.parent / "source.bin" + source.write_bytes(b"metadata-only test input") + service = ImportMetadataService() + service.write(db, service.build_metadata(source, service.calculate_sha256(source), None)) + elif state == "invalid": + with sqlite3.connect(db) as conn: + conn.execute("CREATE TABLE pt_snap_metadata (id INTEGER)") + conn.execute("INSERT INTO pt_snap_metadata VALUES (1)") + elif state == "schema-error": + with sqlite3.connect(db) as conn: + conn.execute("DROP TABLE dictionary") + result = runner.invoke(app, ["overview", str(db), "--json"]) + if state == "schema-error": + assert result.exit_code != 0 and result.stdout == "" + assert json.loads(result.stderr)["error"]["code"] == "DATABASE_SCHEMA_INVALID" + return + assert result.exit_code == 0 + overview = json.loads(result.stdout) + assert overview["ok"] is True # Keep the existing CLI success semantics. + expected = "unavailable" if state == "legacy" else state + assert overview["import_metadata"]["status"] == expected + if state in {"legacy", "invalid"}: + assert overview["import_metadata"]["reason"] == ( + "metadata_missing" if state == "legacy" else "metadata_invalid" + )