Skip to content

fix: restore dump-time torch-npu workspace snapshot correction - #101

Merged
codeasier merged 3 commits into
mainfrom
fix/issue-98-workspace-snapshot
Aug 19, 2026
Merged

codeasier merged 3 commits into
mainfrom
fix/issue-98-workspace-snapshot

Conversation

@codeasier

Copy link
Copy Markdown
Owner

Summary

  • Restore dump-time torch-npu workspace pool occupancy after ignore_inactive_blocks filtering, so workspace_snapshot + segment_alloc + alloc groups are recorded as a single active_allocated block instead of an empty cache.
  • Keep workspace_flag as a fallback; missing segments and size mismatches warn and stop later groups.
  • Replace tests that locked in flag-and-skip behavior with dump-time occupancy assertions.

Fixes #98

Snapshot provenance decision: updated

Test plan

  • pytest tests/snapshot/test_simulate.py
  • pytest (577 passed, 1 xfailed)
  • ruff check . and black --check .

Made with Cursor

@codeasier codeasier left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

交叉审查结果(3 位独立审查者 + 父会话汇总):1 High / 2 Medium / 2 Low / 1 Info,详见行内评论。

Comment thread src/pt_snap_cli/snapshot/PROVENANCE.md Outdated
segment lookup by reusing discovered indices and standard-library keyed
binary search. This is a local performance change against the first-party
runtime; audited upstream source mappings and license terms are unchanged.
- On 2026-08-19, issue #98 restored dump-time torch-npu workspace pool

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

【高】 该条目声称 "restored dump-time torch-npu workspace pool correction",并断言 "Audited upstream source mappings and license terms are unchanged"。但固定上游 MemSnapDump @87ea207372e0985790e1a28dab499dccf3c3b9a4 的 src/memsnapdump/simulate/simulate.py 中并不存在 _adapt_workspace_snapshot()(__init__ 止于 workspace_flag = True)。"restored" 的措辞与该固定源不符,且 issue #98 提到的参考实现仓库 msinsight@ffb1e8187 当前已 404,无法作为映射依据。

PROVENANCE 政策(第 136-141 行)要求每条更新记录准确的 "source or local-change identity"。建议改为:(a) "移植自外部参考实现"并附来源/许可说明,或 (b) 标注为本地重新实现;否则该条目会误导未来的审计者,使其对新方法的来源与许可义务产生错误判断。

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已改。PROVENANCE 不再写 restored-from-audited-upstream:_adapt_workspace_snapshot 不在 MemSnapDump @87ea207 中。条目改为 first-party local reimplementation,行为以 issue #98 的分组约定为准。未增加对 msinsight@ffb1e8187 的源映射。

f"Workspace snapshot at addr {workspace_snapshot.addr} "
f"(stream {workspace_snapshot.stream}) not found in device snapshot"
)
break

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

【中】 组失败(缺 segment / size 不匹配)时 break 会造成部分适配:同一条 trace 中更早的组已被改写为 active_allocated,更晚的组未处理。现有测试 test_workspace_adapt_warns_and_stops_when_segment_missing / test_workspace_adapt_warns_and_skips_when_size_mismatches(test_simulate.py:380-405)均把失败组放在第一位,未覆盖"先成功后失败"的部分应用路径,若未来回归导致 raise 或 continue 将无法被捕获。

建议补充测试:[有效组, 失败组],断言 (i) 第一段已适配为 active_allocated,(ii) 第二段保持 blocks == [],(iii) 仅一条警告,(iv) 总量只反映已适配的组,以锁定 issue #98 记载的 "warn and stop later groups" 行为。

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已补测试 test_workspace_adapt_keeps_earlier_group_when_later_group_is_missing:[有效组, 缺 segment 组],断言第一段已适配、第二段保持空闲、仅一条 warning、总量只含已适配组。生产逻辑未改。

events = snapshot.trace_entries
group_start = 0
while group_start + 2 < len(events):
group = events[group_start : group_start + 3]

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

【中】 循环只校验三个 action 的顺序和 workspace_snapshot.size == segment.total_size,未校验组内 segment_alloc / alloc 的 addr/size 是否与 workspace_snapshot 一致。损坏的三元组(组内 addr/size 不一致)会通过门禁并基于第一条事件适配 segment,随后 replay 会在不一致的 trace 上驱动 allocator。

建议在门禁处增加一致性校验(不一致则 warning + break),例如:group[1].addr == workspace_snapshot.addr and group[1].size == workspace_snapshot.size and group[2].addr == workspace_snapshot.addr and group[2].size == workspace_snapshot.size。

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

本 PR 不加这层门禁。#98 的约定是连续三动作 + 用 T1 的 (addr, stream) 找 segment。组内 addr/size 不一致属于损坏 dump,会改变 issue 未定义的输入行为;replay 仍由 workspace_flag 兜底。

f"does not match segment total_size {existed_seg.total_size}"
)
break
snapshot.total_allocated -= existed_seg.allocated_size

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

【低】 这里手动从 snapshot.total_allocated / total_activated 减去段贡献、清零并清空 blocks,绕过了 snapshot_mutator.detach_block(snapshot_mutator.py:31)已封装的记账路径。当前数据形态下正确(ignore_inactive_blocks=True 保证段贡献为 0),但绕过 mutator 会使总量不变量更容易在后续编辑中破坏(如新增 reserved/active 拆分或 hooker 侧计数器)。

建议重构为遍历 existed_seg.blocks 逐个调用 detach_block,再 attach_block,保持记账逻辑集中在 mutator。

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

不改成 detach_block。ignore_inactive_blocks=True 之后这段通常是 allocated/active 为 0 且 blocks 为空,减总量是 no-op,记账在 attach_block。若 dump 出现 allocated=0 但仍有 leftover block 的不一致数据,先 detach 会把总量打负再贴回去,最终占用变成 0。

requested_size=existed_seg.total_size,
address=existed_seg.address,
state=BlockState.ACTIVE_ALLOCATED,
frames=existed_seg.frames,

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

【低】 新合成的 block 使用 frames=existed_seg.frames(segment 级调用栈)。workspace 场景更忠实的来源应为 workspace_snapshot.frames(组内第一个事件);测试 fixture 中两者均为 [],差异不可见。block2record(entity2record.py:39-54)不序列化 block frames,对 SQLite 输出无影响,但内存消费者 / hooker 会看到空的调用栈。

建议改用 frames=workspace_snapshot.frames 或补充注释说明选择 segment frames 的理由。

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

本 PR 不改 frames 来源。#98 修的是 dump 时刻占用;block2record 不序列化 block frames,现有 fixture 两边都是空列表。继续用 segment frames。

):
self.simulated_allocator_context.workspace_flag = True
# 校正 dump 时刻 workspace 池占用,必须在 ignore_inactive_blocks 之后
self._adapt_workspace_snapshot()

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

【信息】 workspace_flag 在首个事件为 workspace_snapshot 时无条件置位(第 42-46 行),与适配是否成功无关。成功路径上 replay 能正常找到合成的 block,simulated_caching_allocator.py:108-114 的 workspace scenario tolerance 警告不会触发(test_minimal_workspace_input_corrects_dump_time_state_before_replay 已断言)。fallback 仅在失败路径(如不完整标记场景)生效,PR 描述中 "Keep workspace_flag as a fallback" 比字面含义更窄。信息级说明,无需代码改动。

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

同意,无需改代码。workspace_flag 在首事件为 workspace_snapshot 时无条件置位;成功适配后 replay 走合成 block,fallback 只覆盖适配失败或不完整标记。

codeasier and others added 2 commits August 19, 2026 23:49
torch-npu dumps live workspace pools as inactive/zero occupancy, so import
was recording empty dump-time allocated/active state.

Co-authored-by: Cursor <cursoragent@cursor.com>
Earlier tests only failed the first group, so partial application of
later-stop was untested.

Co-authored-by: Cursor <cursoragent@cursor.com>
@codeasier
codeasier force-pushed the fix/issue-98-workspace-snapshot branch from 624d656 to 2e81f8c Compare August 19, 2026 15:50

@codeasier codeasier left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

独立复核轮:ruff check . / black --check . 通过;全量 pytest 676 passed / 1 failed,唯一失败 tests/snapshot/test_sqlite_meta.py::test_type_mapping_supports_pep604_optional 在 main 上同样失败(Python 3.14 strict xpass),与本 PR 无关;.github/scripts/check_snapshot_provenance.py --require-decision 带 PR body 校验通过。

PROVENANCE 高危项与「先成功后失败」部分应用路径测试已妥善解决。新增 2 条行内问题(1 中高 / 1 中)。

附注:PR 描述的 pytest (577 passed, 1 xfailed) 与实际收集数(677)差约 100,建议复核该勾选项。

events = snapshot.trace_entries
group_start = 0
while group_start + 2 < len(events):
group = events[group_start : group_start + 3]

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

【中高】 上一轮关于组内 addr/size 一致性的 medium 问题以「组内不一致属于损坏 dump,replay 仍由 workspace_flag 兜底」为由驳回,但本地实测该理由不成立:

  • 组内 alloc.size > segment total_size:replay 时 free_block 撞到 exist_block.size < alloc_event.size(simulated_caching_allocator.py),ERROR 且 replay() 返回 False。
  • 组内 alloc.addr 落在 segment 范围之外:cannot find segment for block,同样返回 False。

workspace_flag 的容忍分支只覆盖 block not found(simulated_caching_allocator.py:108-114),上述两条路径都绕过它。更关键的是:适配把「段内找不到 block」这个可容忍状态,替换成了「尺寸不匹配 / 段找不到」这两个不可容忍状态——即适配让损坏 dump 的失败模式比适配前更硬(未适配时首个 alloc 会命中 tolerance 并 return True)。

建议二选一:

(a) 在门禁处补组内一致性校验,不一致则 warning + break 回落到未适配路径,与 missing-segment / size-mismatch 的处理保持对称;

(b) 若有意保留 fail-fast,请把 rebuttal 与 docstring 的措辞改为「损坏三元组会使 replay 显式失败」,不要表述为 workspace_flag 覆盖。

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

上一轮用 workspace_flag 兜底的判断有误:tolerance 只覆盖 block not found,exist_block.size < event.size 和 cannot find segment 都会让 replay() 返回 False。已按 (a) 在门禁补组内 addr/size 一致性,不一致则 warning + break,回落到未适配路径。test_workspace_adapt_skips_inconsistent_triplet_so_replay_keeps_tolerance 锁定 alloc.size 过大时 replay 仍为 True。

snapshot.total_activated -= existed_seg.active_size
existed_seg.allocated_size = 0
existed_seg.active_size = 0
existed_seg.blocks = []

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

【中】 这里无条件清空 blocks。若 workspace 段在 dump 中带有 active_allocated 或 active_pending_free block(这类 block 不会被 ignore_inactive_blocks 过滤),会出现三个问题:

  1. 原 block 被静默丢弃,且其 segment_ptr 未清空,留下悬挂引用(snapshot_mutator.detach_block 会正确清空,这条路径绕过了 mutator)。
  2. 全程没有 warning。而 [BUG] torch-npu workspace_snapshot 未校正 dump 时刻 segment 状态 #98 的前提正是「该段只有一个 inactive block」,前提被违反时应当可观测,否则未来 torch-npu dump 形态变化会静默产生错误的 dump 时刻状态。
  3. 实测该真实 block 自身的 alloc 事件在 replay 时落入 workspace scenario tolerance warning,dump 时刻的 block 级归因丢失。总量最终仍收敛到 0,所以现有断言(只看 total_allocated / total_activated)无法捕获。

复现:段 [1000, 4096) 含 active_allocated block [1000, 1024) + inactive block [2048, 2048),trace 为 workspace 三元组 + 内层 block 的 alloc 事件;适配后段内只剩一个 4096 的整段 block,内层 1024 的 block 消失。

建议:existed_seg.blocks 非空时 warning 并 break(保守跳过该组及后续组,与现有失败语义一致),或至少记录被丢弃的 block 数量与地址范围。

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已按保守语义处理:existed_seg.blocks 非空则 warning 并 break,不覆盖仍在的 live block。test_workspace_adapt_skips_when_segment_still_has_live_blocks 锁定 dump 时刻仍是原来的 1024 active_allocated。

Adapting those dumps replaced a tolerable missing-block replay path with
hard size/segment failures, and silently discarded leftover allocations.

Co-authored-by: Cursor <cursoragent@cursor.com>
@codeasier
codeasier merged commit 4398625 into main Aug 19, 2026
6 checks passed
@codeasier
codeasier deleted the fix/issue-98-workspace-snapshot branch August 19, 2026 16:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] torch-npu workspace_snapshot 未校正 dump 时刻 segment 状态

1 participant