Repository navigation
fix: restore dump-time torch-npu workspace snapshot correction #101
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,7 +1,8 @@ | ||
| from logging import Logger | ||
|
|
||
| from ..base import DeviceSnapshot | ||
| from ..base import Block, BlockState, DeviceSnapshot | ||
| from ..util import get_logger | ||
| from . import snapshot_lookup, snapshot_mutator | ||
| from .allocator_context import AllocatorContext | ||
| from .hooker_defs import AllocatorHooker, SimulateHooker | ||
| from .replay_executor import ReplayExecutor | ||
|
|
@@ -46,6 +47,92 @@ def __init__(self, snapshot_dict: dict, device: int, *, _raw_frames: bool = Fals | |
| and self.device_snapshot.trace_entries[0].action == "workspace_snapshot" | ||
| ): | ||
| self.simulated_allocator_context.workspace_flag = True | ||
| # 校正 dump 时刻 workspace 池占用,必须在 ignore_inactive_blocks 之后 | ||
| self._adapt_workspace_snapshot() | ||
|
|
||
| def _adapt_workspace_snapshot(self): | ||
| """Fix dump-time occupancy for torch-npu workspace pools. | ||
|
|
||
| torch-npu prepends consecutive ``workspace_snapshot`` + ``segment_alloc`` | ||
| + ``alloc`` groups for each live workspace pool, but dumps the matching | ||
| segment as one inactive block with allocated/active size 0. After | ||
| ``ignore_inactive_blocks=True`` that block is gone, so replay cannot | ||
| find it and dump-time curves stay at zero. | ||
|
|
||
| Matched groups become a single ``active_allocated`` block covering the | ||
| segment, with allocated/active/device totals equal to the workspace | ||
| size. Missing segments, size mismatches, internally inconsistent | ||
| triplets, or leftover live blocks warn and stop later groups. | ||
| ``workspace_flag`` remains the fallback for those skipped cases. | ||
| """ | ||
| self._loading_logger.info("Recognized workspace events in snapshot, start adapting...") | ||
| snapshot = self.device_snapshot | ||
| events = snapshot.trace_entries | ||
| group_start = 0 | ||
| while group_start + 2 < len(events): | ||
| group = events[group_start : group_start + 3] | ||
|
Owner
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 【中】 循环只校验三个 action 的顺序和 建议在门禁处增加一致性校验(不一致则 warning + break),例如:
Owner
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 本 PR 不加这层门禁。#98 的约定是连续三动作 + 用 T1 的
Owner
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 【中高】 上一轮关于组内 addr/size 一致性的 medium 问题以「组内不一致属于损坏 dump,replay 仍由
建议二选一: (a) 在门禁处补组内一致性校验,不一致则 warning + (b) 若有意保留 fail-fast,请把 rebuttal 与 docstring 的措辞改为「损坏三元组会使 replay 显式失败」,不要表述为
Owner
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 上一轮用 |
||
| if [event.action for event in group] != [ | ||
| "workspace_snapshot", | ||
| "segment_alloc", | ||
| "alloc", | ||
| ]: | ||
| break | ||
| workspace_snapshot, segment_alloc, alloc = group | ||
| if ( | ||
| segment_alloc.addr != workspace_snapshot.addr | ||
| or segment_alloc.size != workspace_snapshot.size | ||
| or alloc.addr != workspace_snapshot.addr | ||
| or alloc.size != workspace_snapshot.size | ||
| ): | ||
| self._loading_logger.warning( | ||
| f"Workspace snapshot triplet at addr {workspace_snapshot.addr} " | ||
| f"(stream {workspace_snapshot.stream}) is internally inconsistent" | ||
| ) | ||
| break | ||
| _, existed_seg = snapshot_lookup.find_segment( | ||
| snapshot, workspace_snapshot.addr, workspace_snapshot.stream | ||
| ) | ||
| if existed_seg is None: | ||
| self._loading_logger.warning( | ||
| f"Workspace snapshot at addr {workspace_snapshot.addr} " | ||
| f"(stream {workspace_snapshot.stream}) not found in device snapshot" | ||
| ) | ||
| break | ||
|
Owner
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 【中】 组失败(缺 segment / size 不匹配)时 建议补充测试:
Owner
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 已补测试 |
||
| if existed_seg.total_size != workspace_snapshot.size: | ||
| self._loading_logger.warning( | ||
| f"Workspace snapshot at addr {workspace_snapshot.addr} " | ||
| f"(stream {workspace_snapshot.stream}) size {workspace_snapshot.size} " | ||
| f"does not match segment total_size {existed_seg.total_size}" | ||
| ) | ||
| break | ||
| if existed_seg.blocks: | ||
| n_live = len(existed_seg.blocks) | ||
| self._loading_logger.warning( | ||
| f"Workspace snapshot at addr {workspace_snapshot.addr} " | ||
| f"(stream {workspace_snapshot.stream}) skipped because the matching " | ||
| f"segment still has {n_live} live block{'s' if n_live != 1 else ''}" | ||
| ) | ||
| break | ||
| snapshot.total_allocated -= existed_seg.allocated_size | ||
|
Owner
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 【低】 这里手动从 建议重构为遍历
Owner
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 不改成 |
||
| snapshot.total_activated -= existed_seg.active_size | ||
| existed_seg.allocated_size = 0 | ||
| existed_seg.active_size = 0 | ||
| existed_seg.blocks = [] | ||
|
Owner
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 【中】 这里无条件清空
复现:段 建议:
Owner
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 已按保守语义处理: |
||
| snapshot_mutator.attach_block( | ||
| snapshot, | ||
| existed_seg, | ||
| Block( | ||
| size=existed_seg.total_size, | ||
| requested_size=existed_seg.total_size, | ||
| address=existed_seg.address, | ||
| state=BlockState.ACTIVE_ALLOCATED, | ||
| frames=existed_seg.frames, | ||
|
Owner
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 【低】 新合成的 block 使用 建议改用
Owner
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 本 PR 不改 frames 来源。#98 修的是 dump 时刻占用; |
||
| segment_ptr=existed_seg, | ||
| ), | ||
| 0, | ||
| ) | ||
| group_start += 3 | ||
| self._loading_logger.info("Finished to adapt workspace snapshot.") | ||
|
|
||
| def register_hooker(self, hooker: SimulateHooker) -> int: | ||
| idx = hash(hooker) | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
【信息】
workspace_flag在首个事件为workspace_snapshot时无条件置位(第 42-46 行),与适配是否成功无关。成功路径上 replay 能正常找到合成的 block,simulated_caching_allocator.py:108-114的workspace scenario tolerance警告不会触发(test_minimal_workspace_input_corrects_dump_time_state_before_replay已断言)。fallback 仅在失败路径(如不完整标记场景)生效,PR 描述中 "Keep workspace_flag as a fallback" 比字面含义更窄。信息级说明,无需代码改动。There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
同意,无需改代码。
workspace_flag在首事件为workspace_snapshot时无条件置位;成功适配后 replay 走合成 block,fallback 只覆盖适配失败或不完整标记。