Repository navigation
fix: restore dump-time torch-npu workspace snapshot correction - #101
Conversation
codeasier
left a comment
There was a problem hiding this comment.
交叉审查结果(3 位独立审查者 + 父会话汇总):1 High / 2 Medium / 2 Low / 1 Info,详见行内评论。
| segment lookup by reusing discovered indices and standard-library keyed | ||
| binary search. This is a local performance change against the first-party | ||
| runtime; audited upstream source mappings and license terms are unchanged. | ||
| - On 2026-08-19, issue #98 restored dump-time torch-npu workspace pool |
There was a problem hiding this comment.
【高】 该条目声称 "restored dump-time torch-npu workspace pool correction",并断言 "Audited upstream source mappings and license terms are unchanged"。但固定上游 MemSnapDump @87ea207372e0985790e1a28dab499dccf3c3b9a4 的 src/memsnapdump/simulate/simulate.py 中并不存在 _adapt_workspace_snapshot()(__init__ 止于 workspace_flag = True)。"restored" 的措辞与该固定源不符,且 issue #98 提到的参考实现仓库 msinsight@ffb1e8187 当前已 404,无法作为映射依据。
PROVENANCE 政策(第 136-141 行)要求每条更新记录准确的 "source or local-change identity"。建议改为:(a) "移植自外部参考实现"并附来源/许可说明,或 (b) 标注为本地重新实现;否则该条目会误导未来的审计者,使其对新方法的来源与许可义务产生错误判断。
There was a problem hiding this comment.
已改。PROVENANCE 不再写 restored-from-audited-upstream:_adapt_workspace_snapshot 不在 MemSnapDump @87ea207 中。条目改为 first-party local reimplementation,行为以 issue #98 的分组约定为准。未增加对 msinsight@ffb1e8187 的源映射。
| f"Workspace snapshot at addr {workspace_snapshot.addr} " | ||
| f"(stream {workspace_snapshot.stream}) not found in device snapshot" | ||
| ) | ||
| break |
There was a problem hiding this comment.
【中】 组失败(缺 segment / size 不匹配)时 break 会造成部分适配:同一条 trace 中更早的组已被改写为 active_allocated,更晚的组未处理。现有测试 test_workspace_adapt_warns_and_stops_when_segment_missing / test_workspace_adapt_warns_and_skips_when_size_mismatches(test_simulate.py:380-405)均把失败组放在第一位,未覆盖"先成功后失败"的部分应用路径,若未来回归导致 raise 或 continue 将无法被捕获。
建议补充测试:[有效组, 失败组],断言 (i) 第一段已适配为 active_allocated,(ii) 第二段保持 blocks == [],(iii) 仅一条警告,(iv) 总量只反映已适配的组,以锁定 issue #98 记载的 "warn and stop later groups" 行为。
There was a problem hiding this comment.
已补测试 test_workspace_adapt_keeps_earlier_group_when_later_group_is_missing:[有效组, 缺 segment 组],断言第一段已适配、第二段保持空闲、仅一条 warning、总量只含已适配组。生产逻辑未改。
| events = snapshot.trace_entries | ||
| group_start = 0 | ||
| while group_start + 2 < len(events): | ||
| group = events[group_start : group_start + 3] |
There was a problem hiding this comment.
【中】 循环只校验三个 action 的顺序和 workspace_snapshot.size == segment.total_size,未校验组内 segment_alloc / alloc 的 addr/size 是否与 workspace_snapshot 一致。损坏的三元组(组内 addr/size 不一致)会通过门禁并基于第一条事件适配 segment,随后 replay 会在不一致的 trace 上驱动 allocator。
建议在门禁处增加一致性校验(不一致则 warning + break),例如:group[1].addr == workspace_snapshot.addr and group[1].size == workspace_snapshot.size and group[2].addr == workspace_snapshot.addr and group[2].size == workspace_snapshot.size。
There was a problem hiding this comment.
本 PR 不加这层门禁。#98 的约定是连续三动作 + 用 T1 的 (addr, stream) 找 segment。组内 addr/size 不一致属于损坏 dump,会改变 issue 未定义的输入行为;replay 仍由 workspace_flag 兜底。
| f"does not match segment total_size {existed_seg.total_size}" | ||
| ) | ||
| break | ||
| snapshot.total_allocated -= existed_seg.allocated_size |
There was a problem hiding this comment.
【低】 这里手动从 snapshot.total_allocated / total_activated 减去段贡献、清零并清空 blocks,绕过了 snapshot_mutator.detach_block(snapshot_mutator.py:31)已封装的记账路径。当前数据形态下正确(ignore_inactive_blocks=True 保证段贡献为 0),但绕过 mutator 会使总量不变量更容易在后续编辑中破坏(如新增 reserved/active 拆分或 hooker 侧计数器)。
建议重构为遍历 existed_seg.blocks 逐个调用 detach_block,再 attach_block,保持记账逻辑集中在 mutator。
There was a problem hiding this comment.
不改成 detach_block。ignore_inactive_blocks=True 之后这段通常是 allocated/active 为 0 且 blocks 为空,减总量是 no-op,记账在 attach_block。若 dump 出现 allocated=0 但仍有 leftover block 的不一致数据,先 detach 会把总量打负再贴回去,最终占用变成 0。
| requested_size=existed_seg.total_size, | ||
| address=existed_seg.address, | ||
| state=BlockState.ACTIVE_ALLOCATED, | ||
| frames=existed_seg.frames, |
There was a problem hiding this comment.
【低】 新合成的 block 使用 frames=existed_seg.frames(segment 级调用栈)。workspace 场景更忠实的来源应为 workspace_snapshot.frames(组内第一个事件);测试 fixture 中两者均为 [],差异不可见。block2record(entity2record.py:39-54)不序列化 block frames,对 SQLite 输出无影响,但内存消费者 / hooker 会看到空的调用栈。
建议改用 frames=workspace_snapshot.frames 或补充注释说明选择 segment frames 的理由。
There was a problem hiding this comment.
本 PR 不改 frames 来源。#98 修的是 dump 时刻占用;block2record 不序列化 block frames,现有 fixture 两边都是空列表。继续用 segment frames。
| ): | ||
| self.simulated_allocator_context.workspace_flag = True | ||
| # 校正 dump 时刻 workspace 池占用,必须在 ignore_inactive_blocks 之后 | ||
| self._adapt_workspace_snapshot() |
There was a problem hiding this comment.
【信息】 workspace_flag 在首个事件为 workspace_snapshot 时无条件置位(第 42-46 行),与适配是否成功无关。成功路径上 replay 能正常找到合成的 block,simulated_caching_allocator.py:108-114 的 workspace scenario tolerance 警告不会触发(test_minimal_workspace_input_corrects_dump_time_state_before_replay 已断言)。fallback 仅在失败路径(如不完整标记场景)生效,PR 描述中 "Keep workspace_flag as a fallback" 比字面含义更窄。信息级说明,无需代码改动。
There was a problem hiding this comment.
同意,无需改代码。workspace_flag 在首事件为 workspace_snapshot 时无条件置位;成功适配后 replay 走合成 block,fallback 只覆盖适配失败或不完整标记。
torch-npu dumps live workspace pools as inactive/zero occupancy, so import was recording empty dump-time allocated/active state. Co-authored-by: Cursor <cursoragent@cursor.com>
Earlier tests only failed the first group, so partial application of later-stop was untested. Co-authored-by: Cursor <cursoragent@cursor.com>
624d656 to
2e81f8c
Compare
codeasier
left a comment
There was a problem hiding this comment.
独立复核轮:ruff check . / black --check . 通过;全量 pytest 676 passed / 1 failed,唯一失败 tests/snapshot/test_sqlite_meta.py::test_type_mapping_supports_pep604_optional 在 main 上同样失败(Python 3.14 strict xpass),与本 PR 无关;.github/scripts/check_snapshot_provenance.py --require-decision 带 PR body 校验通过。
PROVENANCE 高危项与「先成功后失败」部分应用路径测试已妥善解决。新增 2 条行内问题(1 中高 / 1 中)。
附注:PR 描述的 pytest (577 passed, 1 xfailed) 与实际收集数(677)差约 100,建议复核该勾选项。
| events = snapshot.trace_entries | ||
| group_start = 0 | ||
| while group_start + 2 < len(events): | ||
| group = events[group_start : group_start + 3] |
There was a problem hiding this comment.
【中高】 上一轮关于组内 addr/size 一致性的 medium 问题以「组内不一致属于损坏 dump,replay 仍由 workspace_flag 兜底」为由驳回,但本地实测该理由不成立:
- 组内
alloc.size> segmenttotal_size:replay 时free_block撞到exist_block.size < alloc_event.size(simulated_caching_allocator.py),ERROR 且replay()返回 False。 - 组内
alloc.addr落在 segment 范围之外:cannot find segment for block,同样返回 False。
workspace_flag 的容忍分支只覆盖 block not found(simulated_caching_allocator.py:108-114),上述两条路径都绕过它。更关键的是:适配把「段内找不到 block」这个可容忍状态,替换成了「尺寸不匹配 / 段找不到」这两个不可容忍状态——即适配让损坏 dump 的失败模式比适配前更硬(未适配时首个 alloc 会命中 tolerance 并 return True)。
建议二选一:
(a) 在门禁处补组内一致性校验,不一致则 warning + break 回落到未适配路径,与 missing-segment / size-mismatch 的处理保持对称;
(b) 若有意保留 fail-fast,请把 rebuttal 与 docstring 的措辞改为「损坏三元组会使 replay 显式失败」,不要表述为 workspace_flag 覆盖。
There was a problem hiding this comment.
上一轮用 workspace_flag 兜底的判断有误:tolerance 只覆盖 block not found,exist_block.size < event.size 和 cannot find segment 都会让 replay() 返回 False。已按 (a) 在门禁补组内 addr/size 一致性,不一致则 warning + break,回落到未适配路径。test_workspace_adapt_skips_inconsistent_triplet_so_replay_keeps_tolerance 锁定 alloc.size 过大时 replay 仍为 True。
| snapshot.total_activated -= existed_seg.active_size | ||
| existed_seg.allocated_size = 0 | ||
| existed_seg.active_size = 0 | ||
| existed_seg.blocks = [] |
There was a problem hiding this comment.
【中】 这里无条件清空 blocks。若 workspace 段在 dump 中带有 active_allocated 或 active_pending_free block(这类 block 不会被 ignore_inactive_blocks 过滤),会出现三个问题:
- 原 block 被静默丢弃,且其
segment_ptr未清空,留下悬挂引用(snapshot_mutator.detach_block会正确清空,这条路径绕过了 mutator)。 - 全程没有 warning。而 [BUG] torch-npu workspace_snapshot 未校正 dump 时刻 segment 状态 #98 的前提正是「该段只有一个 inactive block」,前提被违反时应当可观测,否则未来 torch-npu dump 形态变化会静默产生错误的 dump 时刻状态。
- 实测该真实 block 自身的
alloc事件在 replay 时落入workspace scenario tolerancewarning,dump 时刻的 block 级归因丢失。总量最终仍收敛到 0,所以现有断言(只看total_allocated/total_activated)无法捕获。
复现:段 [1000, 4096) 含 active_allocated block [1000, 1024) + inactive block [2048, 2048),trace 为 workspace 三元组 + 内层 block 的 alloc 事件;适配后段内只剩一个 4096 的整段 block,内层 1024 的 block 消失。
建议:existed_seg.blocks 非空时 warning 并 break(保守跳过该组及后续组,与现有失败语义一致),或至少记录被丢弃的 block 数量与地址范围。
There was a problem hiding this comment.
已按保守语义处理:existed_seg.blocks 非空则 warning 并 break,不覆盖仍在的 live block。test_workspace_adapt_skips_when_segment_still_has_live_blocks 锁定 dump 时刻仍是原来的 1024 active_allocated。
Adapting those dumps replaced a tolerable missing-block replay path with hard size/segment failures, and silently discarded leftover allocations. Co-authored-by: Cursor <cursoragent@cursor.com>
Summary
ignore_inactive_blocksfiltering, soworkspace_snapshot+segment_alloc+allocgroups are recorded as a singleactive_allocatedblock instead of an empty cache.workspace_flagas a fallback; missing segments and size mismatches warn and stop later groups.Fixes #98
Snapshot provenance decision: updated
Test plan
pytest tests/snapshot/test_simulate.pypytest(577 passed, 1 xfailed)ruff check .andblack --check .Made with Cursor