Skip to content

fix(skills): require iteration evidence before per-step leak conclusions - #170

Merged
codeasier merged 1 commit into
mainfrom
fix/issue-164-iteration-evidence
Oct 3, 2026
Merged

codeasier merged 1 commit into
mainfrom
fix/issue-164-iteration-evidence

Conversation

@codeasier

Copy link
Copy Markdown
Owner

📝 Description

Require trusted iteration evidence before either positive or negative per-step leak conclusions in pt-snap-memory-leak Step 7, the output template, and the verification checklist.

  • Report iteration count, event-ID boundaries, workload code/log/verified marker source, and coverage; unsupported evidence and per-step conclusions become unknown.
  • Handle multiple unmap events per cleanup, multiple cleanups per iteration, and partial capture coverage. Allocator events and cleanup calls cannot directly count steps.
  • Preserve accumulation as a candidate rather than a confirmed leak, and explain why Jaccard set similarity cannot establish a unique byte difference.
  • Synchronize the bundled skill and add an independent four-case iteration-evidence evaluation suite plus static/fixture/grader regression checks.

🔗 Related Issue

Fixes #164

🧪 Testing

  • Tests added/updated
  • Manual testing completed (diff/scope and bundled synchronization reviewed; model evaluation limitation below)

Used /Users/test1/liuyekang/miniconda3/bin/python, worktree-root CWD, isolated HOME, and explicit worktree src plus original site-package paths in PYTHONPATH. Confirmed pt_snap_cli.__file__ resolves inside this worktree. No editable install or shared-environment changes.

  • Fixture provenance first: python -m pytest tests/test_fixture_provenance.py — 7 passed.
  • Focused: python -m pytest tests/skills/test_memory_leak_iterations_contract.py tests/skills/test_memory_leak_contract.py tests/test_bundled_skills.py — 36 passed.
  • Related full checks: python -m pytest tests/skills tests/test_bundled_skills.py — 120 passed.
  • python -m tests.skills validate tests/skills/suites/pt-snap-memory-leak-iterations/suite.yaml — all four cases/branches valid.
  • python -m ruff check . and python -m black --check . — passed.
  • python -m basedpyright --pythonpath /Users/test1/liuyekang/miniconda3/bin/python — 0 errors, 831 informational warnings in unchanged package source.
  • git diff --check — passed.

Evaluation limitation: no live model/agent evaluation was run. Validation covers descriptor/schema loading, generated SQLite fixture queries, and deterministic oracle/grader behavior only; it does not establish actual model adherence. Pytest emitted the installed pytest-asyncio plugin's unset-loop-scope deprecation warning.

✅ Checklist

  • Code follows project guidelines
  • Documentation updated (skill authoring source and bundled copy)
  • No new warnings introduced (existing environment/type-check warnings noted above)
  • Changes tested locally

Snapshot Runtime Provenance

  • Snapshot provenance evaluation completed when snapshot runtime files change
    Snapshot provenance decision: no-update
    Snapshot provenance no-update reason: Changes are limited to skill instructions and declarative synthetic skill evaluations/tests; no snapshot runtime files or executable snapshot fixtures changed.

📸 Screenshots (if applicable)

Not applicable.

📋 Additional Notes

@codeasier
codeasier merged commit 729fe5b into main Oct 3, 2026
6 checks passed
@codeasier
codeasier deleted the fix/issue-164-iteration-evidence branch October 3, 2026 19:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Determine the iteration count before classifying a per-step leak

1 participant