Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 36 additions & 3 deletions skills/pt-snap-memory-leak/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -183,6 +183,35 @@ This baseline describes successfully freed blocks, bucketed by `freeEventId - al

### 7. Classify findings conservatively

Before any positive or negative per-step / per-iteration conclusion (including
"not a per-step leak"), establish and report iteration evidence:

- State the trusted iteration count, boundaries mapped to event IDs, source,
and coverage range. Use workload code, logs, or an explicit verified marker
convention to establish the mapping; allocator event counts, waveform minima,
and cleanup counts are not iteration counts.
- `segment_unmap` is only a conditional supporting clue. One `empty_cache`
call can emit multiple unmap events; one iteration can perform multiple
cleanups, or none. Group events and cleanup calls only according to the
verified workload mapping, never by assuming one event or cleanup equals
one step. Do not directly count `segment_unmap` events as steps.
- Check capture start/end and missing markers against that source. Separate
fully covered iterations from partial iterations and uncaptured work; do not
extrapolate a captured count to the whole run. If count, boundaries, source,
or coverage cannot be established, report the unsupported fields and the
per-step conclusion as `unknown`. A verified complete subrange may support
a conclusion scoped only to that subrange, with the rest explicitly unknown.
- Within verified comparable boundaries, correlate allocation identities and
release events with live bytes after expected cleanup. Three surviving blocks
over three verified iterations can support accumulation; three blocks alone
cannot establish a fixed per-step population or rule out accumulation. Keep
observed per-iteration accumulation a candidate, not a confirmed leak; check
application ownership and expected lifetime before assigning a category.
- Jaccard similarity of `(action, callstackId)` sets is only supporting evidence
of similar event kinds. Sets discard multiplicity, ordering, allocation
identity, and byte sizes; even high similarity cannot prove that unreturned
bytes are the only per-iteration difference.

Use these result categories:

- `strong leak candidate`: dynamic live bytes grow across comparable captures or phases, persist after expected cleanup, and have ownership evidence. Do not use this category from end-of-trace survival alone.
Expand All @@ -202,9 +231,10 @@ Report results in this order:
3. `End-of-trace evidence`: dynamic candidate count, dynamic bytes by callstack, static bytes, and representative blocks.
4. `Peak-occupancy evidence`: callstack bytes and block counts at the active peak versus the final event, identity-matched blocks if any, and static plus `[preexisting live]` memory that carries no per-callstack attribution.
5. `Lifecycle evidence`: representative alloc, free-requested, and free-completed event IDs, with ambiguities called out.
6. `Findings`: category, confidence, evidence, inference, and affected callstack or allocation family.
7. `Unknowns`: missing timing, ownership, capture-boundary, callstack, or repeated-capture evidence.
8. `Suggested validation`: targeted experiments that could confirm or reject each leading hypothesis.
6. `Iteration evidence`: trusted iteration count, event-ID boundaries, workload code/log/verified marker source, and coverage range; distinguish complete and partial iterations. Report unsupported fields and per-step conclusions as `unknown` before any positive or negative per-step finding. Include scoped per-iteration live-byte/lifecycle evidence when available.
7. `Findings`: category, confidence, evidence, inference, and affected callstack or allocation family; per-iteration accumulation remains a candidate, not a confirmed leak.
8. `Unknowns`: missing timing, ownership, capture-boundary, callstack, repeated-capture, or iteration-mapping evidence, including uncaptured work.
9. `Suggested validation`: targeted experiments that could confirm or reject each leading hypothesis, including workload logs or explicit marker conventions when iteration mapping is unknown.

Useful validation experiments include repeated snapshots at equivalent workload milestones, extending tracing beyond expected cleanup, explicitly releasing suspected application references, synchronizing the device before the capture ends, and comparing behavior before and after allocator cache cleanup. Explain that cache cleanup can change reserved memory without proving that application references were released.

Expand Down Expand Up @@ -238,5 +268,8 @@ Useful validation experiments include repeated snapshots at equivalent workload
- Representative lifecycle events were checked for address reuse and ambiguous pairing.
- Dynamic block state was not used as leak evidence.
- Event-ID distances were not presented as time durations.
- Every positive or negative per-step conclusion was preceded by a trusted iteration count, event-ID boundaries, source, and coverage range; otherwise the unsupported fields and conclusion were `unknown`.
- Complete iterations were separated from partial or uncaptured work; multiple allocator events or cleanup calls per iteration were reconciled with workload code, logs, or a verified marker convention instead of counted as steps.
- Per-iteration accumulation remained a candidate, not a confirmed leak, and Jaccard set similarity was not used to claim a unique byte difference.
- Every finding has a conservative category and confidence level.
- Unknowns and validation experiments are included.
39 changes: 36 additions & 3 deletions src/pt_snap_cli/bundled_skills/pt-snap-memory-leak/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -183,6 +183,35 @@ This baseline describes successfully freed blocks, bucketed by `freeEventId - al

### 7. Classify findings conservatively

Before any positive or negative per-step / per-iteration conclusion (including
"not a per-step leak"), establish and report iteration evidence:

- State the trusted iteration count, boundaries mapped to event IDs, source,
and coverage range. Use workload code, logs, or an explicit verified marker
convention to establish the mapping; allocator event counts, waveform minima,
and cleanup counts are not iteration counts.
- `segment_unmap` is only a conditional supporting clue. One `empty_cache`
call can emit multiple unmap events; one iteration can perform multiple
cleanups, or none. Group events and cleanup calls only according to the
verified workload mapping, never by assuming one event or cleanup equals
one step. Do not directly count `segment_unmap` events as steps.
- Check capture start/end and missing markers against that source. Separate
fully covered iterations from partial iterations and uncaptured work; do not
extrapolate a captured count to the whole run. If count, boundaries, source,
or coverage cannot be established, report the unsupported fields and the
per-step conclusion as `unknown`. A verified complete subrange may support
a conclusion scoped only to that subrange, with the rest explicitly unknown.
- Within verified comparable boundaries, correlate allocation identities and
release events with live bytes after expected cleanup. Three surviving blocks
over three verified iterations can support accumulation; three blocks alone
cannot establish a fixed per-step population or rule out accumulation. Keep
observed per-iteration accumulation a candidate, not a confirmed leak; check
application ownership and expected lifetime before assigning a category.
- Jaccard similarity of `(action, callstackId)` sets is only supporting evidence
of similar event kinds. Sets discard multiplicity, ordering, allocation
identity, and byte sizes; even high similarity cannot prove that unreturned
bytes are the only per-iteration difference.

Use these result categories:

- `strong leak candidate`: dynamic live bytes grow across comparable captures or phases, persist after expected cleanup, and have ownership evidence. Do not use this category from end-of-trace survival alone.
Expand All @@ -202,9 +231,10 @@ Report results in this order:
3. `End-of-trace evidence`: dynamic candidate count, dynamic bytes by callstack, static bytes, and representative blocks.
4. `Peak-occupancy evidence`: callstack bytes and block counts at the active peak versus the final event, identity-matched blocks if any, and static plus `[preexisting live]` memory that carries no per-callstack attribution.
5. `Lifecycle evidence`: representative alloc, free-requested, and free-completed event IDs, with ambiguities called out.
6. `Findings`: category, confidence, evidence, inference, and affected callstack or allocation family.
7. `Unknowns`: missing timing, ownership, capture-boundary, callstack, or repeated-capture evidence.
8. `Suggested validation`: targeted experiments that could confirm or reject each leading hypothesis.
6. `Iteration evidence`: trusted iteration count, event-ID boundaries, workload code/log/verified marker source, and coverage range; distinguish complete and partial iterations. Report unsupported fields and per-step conclusions as `unknown` before any positive or negative per-step finding. Include scoped per-iteration live-byte/lifecycle evidence when available.
7. `Findings`: category, confidence, evidence, inference, and affected callstack or allocation family; per-iteration accumulation remains a candidate, not a confirmed leak.
8. `Unknowns`: missing timing, ownership, capture-boundary, callstack, repeated-capture, or iteration-mapping evidence, including uncaptured work.
9. `Suggested validation`: targeted experiments that could confirm or reject each leading hypothesis, including workload logs or explicit marker conventions when iteration mapping is unknown.

Useful validation experiments include repeated snapshots at equivalent workload milestones, extending tracing beyond expected cleanup, explicitly releasing suspected application references, synchronizing the device before the capture ends, and comparing behavior before and after allocator cache cleanup. Explain that cache cleanup can change reserved memory without proving that application references were released.

Expand Down Expand Up @@ -238,5 +268,8 @@ Useful validation experiments include repeated snapshots at equivalent workload
- Representative lifecycle events were checked for address reuse and ambiguous pairing.
- Dynamic block state was not used as leak evidence.
- Event-ID distances were not presented as time durations.
- Every positive or negative per-step conclusion was preceded by a trusted iteration count, event-ID boundaries, source, and coverage range; otherwise the unsupported fields and conclusion were `unknown`.
- Complete iterations were separated from partial or uncaptured work; multiple allocator events or cleanup calls per iteration were reconciled with workload code, logs, or a verified marker convention instead of counted as steps.
- Per-iteration accumulation remained a candidate, not a confirmed leak, and Jaccard set similarity was not used to claim a unique byte difference.
- Every finding has a conservative category and confidence level.
- Unknowns and validation experiments are included.
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
schema_version: 1
kind: skill-eval-case
id: incomplete-capture
title: Partial capture cannot establish whole-run iteration counts
tags: [iterations, partial-capture, unknown]
covers: [iterations.incomplete_capture]
task:
initial_user: >
Diagnose /fixtures/iterations.db device 0. A verified workload log says
events 1-3 are only a trailing fragment of an iteration whose start was
not captured. Inclusive ranges [4,6] and [7,9] are two complete iterations
through their expected cleanup. Their step numbers and the total run
length are unavailable. Tracing ends at event 9 while the workload
continues. There is no ownership evidence for demo.py:11. Can you infer
the whole-run count or rule out per-step leakage from three surviving
blocks? Separate any fully covered subrange evidence from unknown
whole-run conclusions, and request missing capture/ownership evidence.
fixture:
builder: synthetic-snapshotdb
definition: fixtures/accumulation.yaml
mount_path: /fixtures/iterations.db
read_only: true
expected_tools:
actions:
- id: orient
operation: pt_snap.overview
match: {database: /fixtures/iterations.db, json: true}
- id: candidates
operation: pt_snap.query
match: {database: /fixtures/iterations.db, device: 0, template: leak_detection}
expect_output: {allocEventId: 7, size: 41943040}
- id: events
operation: pt_snap.query
match: {database: /fixtures/iterations.db, device: 0, template: event}
expect_output: {id: 9, action: 1}
partial_order:
- {before: orient, after: candidates}
- {before: orient, after: events}
forbidden: []
oracle:
facts:
whole_run_iteration_count: unknown
whole_run_per_step_conclusion: unknown
complete_iterations: 2
partial_iterations: 1
iteration_boundaries: [[4, 6], [7, 9]]
iteration_source: verified workload log
coverage_range: [4, 9]
subrange_per_iteration_candidate_bytes: 41943040
end_live_candidate_bytes: 125829120
confirmed_leak: false
result:
allowed_classifications: [inconclusive]
prohibited_classifications: [strong leak candidate]
required_unknowns: [whole-run iteration count, uncaptured work, application ownership, eventual release beyond capture]
prohibited_claims: [not a per-step leak, whole run contains three iterations]
required_evidence:
- {claim: end_live_candidate_bytes, produced_by: candidates}
- {claim: subrange_per_iteration_candidate_bytes, produced_by: events}
scoring:
mandatory_objectives: [safety.read_only, tools.correct_path, evidence.factual_accuracy, evidence.traceability, diagnosis.classification, diagnosis.uncertainty]
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
schema_version: 1
kind: skill-eval-case
id: multiple-cleanups
title: Multiple cleanup calls in each verified iteration
tags: [iterations, cleanup-multiplicity]
covers: [iterations.multiple_cleanups]
task:
initial_user: >
Diagnose /fixtures/iterations.db device 0. The verified workload log maps
complete iterations 1, 2, 3 to inclusive event ranges [1,3], [4,6], [7,9],
covering their starts through cleanup with no missing events. Reviewed
code and logs establish TWO empty_cache calls per iteration: each emits
one unmap, at events 2/3, 5/6, and 8/9. A temporary sample appended at
demo.py:11 remains in a user-owned list intended to be cleared each round.
Are the six cleanup calls six steps, so three surviving blocks rule out
per-step growth? Establish count, boundaries, source and coverage before
classifying. Retain the distinction between candidates and confirmed leaks.
fixture:
builder: synthetic-snapshotdb
definition: fixtures/accumulation.yaml
mount_path: /fixtures/iterations.db
read_only: true
expected_tools:
actions:
- id: orient
operation: pt_snap.overview
match: {database: /fixtures/iterations.db, json: true}
- id: candidates
operation: pt_snap.query
match: {database: /fixtures/iterations.db, device: 0, template: leak_detection}
expect_output: {allocEventId: 7, size: 41943040}
- id: events
operation: pt_snap.query
match: {database: /fixtures/iterations.db, device: 0, template: event}
expect_output: {id: 9, action: 1}
partial_order:
- {before: orient, after: candidates}
- {before: orient, after: events}
forbidden: []
oracle:
facts:
iteration_count: 3
iteration_boundaries: [[1, 3], [4, 6], [7, 9]]
iteration_source: verified workload log
coverage_range: [1, 9]
complete_iterations: 3
cleanup_calls: 6
allocator_events: 9
unmap_events: 6
end_live_candidate_bytes: 125829120
per_iteration_candidate_bytes: 41943040
confirmed_leak: false
result:
allowed_classifications: [strong leak candidate]
prohibited_classifications: [inconclusive]
required_unknowns: [eventual release beyond capture]
prohibited_claims: [not a per-step leak, six iterations]
required_evidence:
- {claim: end_live_candidate_bytes, produced_by: candidates}
- {claim: unmap_events, produced_by: events}
scoring:
mandatory_objectives: [safety.read_only, tools.correct_path, evidence.factual_accuracy, evidence.traceability, diagnosis.classification, diagnosis.uncertainty]
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
schema_version: 1
kind: skill-eval-case
id: no-verified-markers
title: Repeating allocator patterns without a verified step mapping
tags: [iterations, unknown, missing-markers]
covers: [iterations.no_verified_markers]
task:
initial_user: >
Diagnose /fixtures/iterations.db device 0. No workload code, logs, or
verified iteration marker convention is available. Capture completeness
is unknown. An analyst says the repeating allocation and unmap pattern
proves three iterations; another guesses eight from waveform minima and
says three survivors disprove per-step leakage. Can either conclusion be
supported? Report iteration count, boundaries, source and coverage, plus
observable retention. Do not invent step boundaries from allocator events.
fixture:
builder: synthetic-snapshotdb
definition: fixtures/accumulation.yaml
mount_path: /fixtures/iterations.db
read_only: true
expected_tools:
actions:
- id: orient
operation: pt_snap.overview
match: {database: /fixtures/iterations.db, json: true}
- id: candidates
operation: pt_snap.query
match: {database: /fixtures/iterations.db, device: 0, template: leak_detection}
expect_output: {allocEventId: 7, size: 41943040}
- id: events
operation: pt_snap.query
match: {database: /fixtures/iterations.db, device: 0, template: event}
expect_output: {id: 9, action: 1}
partial_order:
- {before: orient, after: candidates}
- {before: orient, after: events}
forbidden: []
oracle:
facts:
iteration_count: unknown
iteration_boundaries: unknown
iteration_source: unknown
iteration_coverage: unknown
per_step_conclusion: unknown
cleanup_calls: unknown
allocator_events: 9
unmap_events: 6
end_live_candidate_bytes: 125829120
confirmed_leak: false
result:
allowed_classifications: [inconclusive]
prohibited_classifications: [strong leak candidate]
required_unknowns: [iteration count, iteration boundaries, iteration source, iteration coverage, application ownership]
prohibited_claims: [not a per-step leak, per-step leak confirmed]
required_evidence:
- {claim: end_live_candidate_bytes, produced_by: candidates}
- {claim: unmap_events, produced_by: events}
scoring:
mandatory_objectives: [safety.read_only, tools.correct_path, evidence.factual_accuracy, evidence.traceability, diagnosis.classification, diagnosis.uncertainty]
Loading
Loading