Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions src/pt_snap_cli/snapshot/PROVENANCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -123,6 +123,17 @@ The upstream standalone split frontend is deliberately excluded, not deferred:
objects and preserves source frame dictionaries through synthetic segments.
This is a local performance change; audited upstream mappings and licensing
evidence are unchanged.
- On 2026-08-19, issue #98 added a first-party dump-time torch-npu workspace
pool correction after inactive-block filtering. `_adapt_workspace_snapshot`
is not present in audited MemSnapDump @87ea207; this is a local
reimplementation of the grouping behavior specified in #98. Consecutive
`workspace_snapshot` + `segment_alloc` + `alloc` groups rewrite the
matching dump-time segment to a single `active_allocated` block and
update device allocated/active totals. Size mismatches, missing
segments, internally inconsistent triplets, and leftover live blocks
warn and stop later groups. `workspace_flag` remains a
fallback for unmatched workspace traces. Audited upstream source
mappings and license terms are unchanged.

## Migration toolchain

Expand Down
89 changes: 88 additions & 1 deletion src/pt_snap_cli/snapshot/simulate/simulate.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
from logging import Logger

from ..base import DeviceSnapshot
from ..base import Block, BlockState, DeviceSnapshot
from ..util import get_logger
from . import snapshot_lookup, snapshot_mutator
from .allocator_context import AllocatorContext
from .hooker_defs import AllocatorHooker, SimulateHooker
from .replay_executor import ReplayExecutor
Expand Down Expand Up @@ -46,6 +47,92 @@ def __init__(self, snapshot_dict: dict, device: int, *, _raw_frames: bool = Fals
and self.device_snapshot.trace_entries[0].action == "workspace_snapshot"
):
self.simulated_allocator_context.workspace_flag = True
# 校正 dump 时刻 workspace 池占用,必须在 ignore_inactive_blocks 之后
self._adapt_workspace_snapshot()

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

【信息】 workspace_flag 在首个事件为 workspace_snapshot 时无条件置位(第 42-46 行),与适配是否成功无关。成功路径上 replay 能正常找到合成的 block,simulated_caching_allocator.py:108-114 的 workspace scenario tolerance 警告不会触发(test_minimal_workspace_input_corrects_dump_time_state_before_replay 已断言)。fallback 仅在失败路径(如不完整标记场景)生效,PR 描述中 "Keep workspace_flag as a fallback" 比字面含义更窄。信息级说明,无需代码改动。

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

同意,无需改代码。workspace_flag 在首事件为 workspace_snapshot 时无条件置位;成功适配后 replay 走合成 block,fallback 只覆盖适配失败或不完整标记。


def _adapt_workspace_snapshot(self):
"""Fix dump-time occupancy for torch-npu workspace pools.

torch-npu prepends consecutive ``workspace_snapshot`` + ``segment_alloc``
+ ``alloc`` groups for each live workspace pool, but dumps the matching
segment as one inactive block with allocated/active size 0. After
``ignore_inactive_blocks=True`` that block is gone, so replay cannot
find it and dump-time curves stay at zero.

Matched groups become a single ``active_allocated`` block covering the
segment, with allocated/active/device totals equal to the workspace
size. Missing segments, size mismatches, internally inconsistent
triplets, or leftover live blocks warn and stop later groups.
``workspace_flag`` remains the fallback for those skipped cases.
"""
self._loading_logger.info("Recognized workspace events in snapshot, start adapting...")
snapshot = self.device_snapshot
events = snapshot.trace_entries
group_start = 0
while group_start + 2 < len(events):
group = events[group_start : group_start + 3]

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

【中】 循环只校验三个 action 的顺序和 workspace_snapshot.size == segment.total_size,未校验组内 segment_alloc / alloc 的 addr/size 是否与 workspace_snapshot 一致。损坏的三元组(组内 addr/size 不一致)会通过门禁并基于第一条事件适配 segment,随后 replay 会在不一致的 trace 上驱动 allocator。

建议在门禁处增加一致性校验(不一致则 warning + break),例如:group[1].addr == workspace_snapshot.addr and group[1].size == workspace_snapshot.size and group[2].addr == workspace_snapshot.addr and group[2].size == workspace_snapshot.size。

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

本 PR 不加这层门禁。#98 的约定是连续三动作 + 用 T1 的 (addr, stream) 找 segment。组内 addr/size 不一致属于损坏 dump,会改变 issue 未定义的输入行为;replay 仍由 workspace_flag 兜底。

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

【中高】 上一轮关于组内 addr/size 一致性的 medium 问题以「组内不一致属于损坏 dump,replay 仍由 workspace_flag 兜底」为由驳回,但本地实测该理由不成立:

  • 组内 alloc.size > segment total_size:replay 时 free_block 撞到 exist_block.size < alloc_event.size(simulated_caching_allocator.py),ERROR 且 replay() 返回 False。
  • 组内 alloc.addr 落在 segment 范围之外:cannot find segment for block,同样返回 False。

workspace_flag 的容忍分支只覆盖 block not found(simulated_caching_allocator.py:108-114),上述两条路径都绕过它。更关键的是:适配把「段内找不到 block」这个可容忍状态,替换成了「尺寸不匹配 / 段找不到」这两个不可容忍状态——即适配让损坏 dump 的失败模式比适配前更硬(未适配时首个 alloc 会命中 tolerance 并 return True)。

建议二选一:

(a) 在门禁处补组内一致性校验,不一致则 warning + break 回落到未适配路径,与 missing-segment / size-mismatch 的处理保持对称;

(b) 若有意保留 fail-fast,请把 rebuttal 与 docstring 的措辞改为「损坏三元组会使 replay 显式失败」,不要表述为 workspace_flag 覆盖。

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

上一轮用 workspace_flag 兜底的判断有误:tolerance 只覆盖 block not found,exist_block.size < event.size 和 cannot find segment 都会让 replay() 返回 False。已按 (a) 在门禁补组内 addr/size 一致性,不一致则 warning + break,回落到未适配路径。test_workspace_adapt_skips_inconsistent_triplet_so_replay_keeps_tolerance 锁定 alloc.size 过大时 replay 仍为 True。

if [event.action for event in group] != [
"workspace_snapshot",
"segment_alloc",
"alloc",
]:
break
workspace_snapshot, segment_alloc, alloc = group
if (
segment_alloc.addr != workspace_snapshot.addr
or segment_alloc.size != workspace_snapshot.size
or alloc.addr != workspace_snapshot.addr
or alloc.size != workspace_snapshot.size
):
self._loading_logger.warning(
f"Workspace snapshot triplet at addr {workspace_snapshot.addr} "
f"(stream {workspace_snapshot.stream}) is internally inconsistent"
)
break
_, existed_seg = snapshot_lookup.find_segment(
snapshot, workspace_snapshot.addr, workspace_snapshot.stream
)
if existed_seg is None:
self._loading_logger.warning(
f"Workspace snapshot at addr {workspace_snapshot.addr} "
f"(stream {workspace_snapshot.stream}) not found in device snapshot"
)
break

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

【中】 组失败(缺 segment / size 不匹配)时 break 会造成部分适配:同一条 trace 中更早的组已被改写为 active_allocated,更晚的组未处理。现有测试 test_workspace_adapt_warns_and_stops_when_segment_missing / test_workspace_adapt_warns_and_skips_when_size_mismatches(test_simulate.py:380-405)均把失败组放在第一位,未覆盖"先成功后失败"的部分应用路径,若未来回归导致 raise 或 continue 将无法被捕获。

建议补充测试:[有效组, 失败组],断言 (i) 第一段已适配为 active_allocated,(ii) 第二段保持 blocks == [],(iii) 仅一条警告,(iv) 总量只反映已适配的组,以锁定 issue #98 记载的 "warn and stop later groups" 行为。

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已补测试 test_workspace_adapt_keeps_earlier_group_when_later_group_is_missing:[有效组, 缺 segment 组],断言第一段已适配、第二段保持空闲、仅一条 warning、总量只含已适配组。生产逻辑未改。

if existed_seg.total_size != workspace_snapshot.size:
self._loading_logger.warning(
f"Workspace snapshot at addr {workspace_snapshot.addr} "
f"(stream {workspace_snapshot.stream}) size {workspace_snapshot.size} "
f"does not match segment total_size {existed_seg.total_size}"
)
break
if existed_seg.blocks:
n_live = len(existed_seg.blocks)
self._loading_logger.warning(
f"Workspace snapshot at addr {workspace_snapshot.addr} "
f"(stream {workspace_snapshot.stream}) skipped because the matching "
f"segment still has {n_live} live block{'s' if n_live != 1 else ''}"
)
break
snapshot.total_allocated -= existed_seg.allocated_size

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

【低】 这里手动从 snapshot.total_allocated / total_activated 减去段贡献、清零并清空 blocks,绕过了 snapshot_mutator.detach_block(snapshot_mutator.py:31)已封装的记账路径。当前数据形态下正确(ignore_inactive_blocks=True 保证段贡献为 0),但绕过 mutator 会使总量不变量更容易在后续编辑中破坏(如新增 reserved/active 拆分或 hooker 侧计数器)。

建议重构为遍历 existed_seg.blocks 逐个调用 detach_block,再 attach_block,保持记账逻辑集中在 mutator。

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

不改成 detach_block。ignore_inactive_blocks=True 之后这段通常是 allocated/active 为 0 且 blocks 为空,减总量是 no-op,记账在 attach_block。若 dump 出现 allocated=0 但仍有 leftover block 的不一致数据,先 detach 会把总量打负再贴回去,最终占用变成 0。

snapshot.total_activated -= existed_seg.active_size
existed_seg.allocated_size = 0
existed_seg.active_size = 0
existed_seg.blocks = []

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

【中】 这里无条件清空 blocks。若 workspace 段在 dump 中带有 active_allocated 或 active_pending_free block(这类 block 不会被 ignore_inactive_blocks 过滤),会出现三个问题:

  1. 原 block 被静默丢弃,且其 segment_ptr 未清空,留下悬挂引用(snapshot_mutator.detach_block 会正确清空,这条路径绕过了 mutator)。
  2. 全程没有 warning。而 [BUG] torch-npu workspace_snapshot 未校正 dump 时刻 segment 状态 #98 的前提正是「该段只有一个 inactive block」,前提被违反时应当可观测,否则未来 torch-npu dump 形态变化会静默产生错误的 dump 时刻状态。
  3. 实测该真实 block 自身的 alloc 事件在 replay 时落入 workspace scenario tolerance warning,dump 时刻的 block 级归因丢失。总量最终仍收敛到 0,所以现有断言(只看 total_allocated / total_activated)无法捕获。

复现:段 [1000, 4096) 含 active_allocated block [1000, 1024) + inactive block [2048, 2048),trace 为 workspace 三元组 + 内层 block 的 alloc 事件;适配后段内只剩一个 4096 的整段 block,内层 1024 的 block 消失。

建议:existed_seg.blocks 非空时 warning 并 break(保守跳过该组及后续组,与现有失败语义一致),或至少记录被丢弃的 block 数量与地址范围。

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已按保守语义处理:existed_seg.blocks 非空则 warning 并 break,不覆盖仍在的 live block。test_workspace_adapt_skips_when_segment_still_has_live_blocks 锁定 dump 时刻仍是原来的 1024 active_allocated。

snapshot_mutator.attach_block(
snapshot,
existed_seg,
Block(
size=existed_seg.total_size,
requested_size=existed_seg.total_size,
address=existed_seg.address,
state=BlockState.ACTIVE_ALLOCATED,
frames=existed_seg.frames,

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

【低】 新合成的 block 使用 frames=existed_seg.frames(segment 级调用栈)。workspace 场景更忠实的来源应为 workspace_snapshot.frames(组内第一个事件);测试 fixture 中两者均为 [],差异不可见。block2record(entity2record.py:39-54)不序列化 block frames,对 SQLite 输出无影响,但内存消费者 / hooker 会看到空的调用栈。

建议改用 frames=workspace_snapshot.frames 或补充注释说明选择 segment frames 的理由。

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

本 PR 不改 frames 来源。#98 修的是 dump 时刻占用;block2record 不序列化 block frames,现有 fixture 两边都是空列表。继续用 segment frames。

segment_ptr=existed_seg,
),
0,
)
group_start += 3
self._loading_logger.info("Finished to adapt workspace snapshot.")

def register_hooker(self, hooker: SimulateHooker) -> int:
idx = hash(hooker)
Expand Down
234 changes: 232 additions & 2 deletions tests/snapshot/test_simulate.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import logging
import os
import unittest
from collections import Counter
Expand Down Expand Up @@ -169,15 +170,99 @@ def make_snapshot_dict(action="alloc", with_segment=False):
return payload


def _inactive_workspace_segment(addr, size, stream):
return {
"address": addr,
"total_size": size,
"stream": stream,
"segment_type": "small",
"allocated_size": 0,
"active_size": 0,
"device": 0,
"is_expandable": False,
"frames": [],
"blocks": [
{
"size": size,
"requested_size": size,
"state": "inactive",
"address": addr,
"frames": [],
}
],
}


def _workspace_triplet(addr, size, stream):
return [
{
"action": "workspace_snapshot",
"addr": addr,
"size": size,
"stream": stream,
"frames": [],
},
{
"action": "segment_alloc",
"addr": addr,
"size": size,
"stream": stream,
"frames": [],
},
{"action": "alloc", "addr": addr, "size": size, "stream": stream, "frames": []},
]


def make_torch_npu_workspace_snapshot(
*, addr=1000, size=4096, stream=1, segment_size=None, extra_segments=None, extra_events=None
):
segment_size = size if segment_size is None else segment_size
segments = [_inactive_workspace_segment(addr, segment_size, stream)]
if extra_segments:
segments.extend(extra_segments)
events = _workspace_triplet(addr, size, stream)
if extra_events:
events.extend(extra_events)
return {"segments": segments, "device_traces": [events]}


def _segment_by_addr_stream(snapshot, addr, stream):
matches = [
segment
for segment in snapshot.device_snapshot.segments
if segment.address == addr and segment.stream == stream
]
assert len(matches) == 1
return matches[0]


def assert_dump_time_workspace_pool(snapshot, *, addr=1000, size=4096, stream=1):
assert snapshot.simulated_allocator_context.workspace_flag is True
segment = _segment_by_addr_stream(snapshot, addr, stream)
assert segment.allocated_size == size
assert segment.active_size == size
assert len(segment.blocks) == 1
block = segment.blocks[0]
assert block.state == BlockState.ACTIVE_ALLOCATED
assert block.address == addr
assert block.size == size
assert_valid_snapshot(snapshot.device_snapshot)


def test_simulate_device_snapshot_raises_on_empty_snapshot():
with pytest.raises(RuntimeError):
SimulateDeviceSnapshot({}, 0)


def test_simulate_device_snapshot_sets_workspace_flag_from_first_event():
snapshot = SimulateDeviceSnapshot(make_snapshot_dict("workspace_snapshot"), 0)
snapshot = SimulateDeviceSnapshot(make_torch_npu_workspace_snapshot(), 0)
device = snapshot.device_snapshot

assert snapshot.simulated_allocator_context.workspace_flag is True
assert_dump_time_workspace_pool(snapshot)
assert device.total_allocated == 4096
assert device.total_activated == 4096
assert device.total_reserved == 4096


def test_simulate_register_unregister_hookers_and_allocator_hookers():
Expand Down Expand Up @@ -253,9 +338,154 @@ def test_multi_device_fixture_keeps_device_state_isolated():
assert snapshot_observation(device_zero) != snapshot_observation(device_one)


def test_minimal_workspace_input_only_establishes_tolerance_flag_and_skip_behavior():
def test_minimal_workspace_input_corrects_dump_time_state_before_replay(caplog):
snapshot = SimulateDeviceSnapshot(make_torch_npu_workspace_snapshot(), 0)

assert_dump_time_workspace_pool(snapshot)
with caplog.at_level(logging.WARNING, logger="ALLOCATOR"):
assert snapshot.replay() is True
assert "workspace scenario tolerance" not in caplog.text
assert snapshot.device_snapshot.trace_entries == []
assert snapshot.device_snapshot.segments == []
assert snapshot.device_snapshot.total_allocated == 0
assert snapshot.device_snapshot.total_activated == 0
assert snapshot.device_snapshot.total_reserved == 0


def test_incomplete_workspace_marker_keeps_tolerance_flag_and_skip_behavior():
snapshot = SimulateDeviceSnapshot(make_snapshot_dict("workspace_snapshot"), 0)

assert snapshot.simulated_allocator_context.workspace_flag is True
assert snapshot.replay() is True
assert snapshot.device_snapshot.trace_entries == []


def test_workspace_adapt_multiple_stream_groups_before_replay():
data = {
"segments": [
_inactive_workspace_segment(1000, 4096, 1),
_inactive_workspace_segment(10000, 8192, 2),
],
"device_traces": [_workspace_triplet(1000, 4096, 1) + _workspace_triplet(10000, 8192, 2)],
}
snapshot = SimulateDeviceSnapshot(data, 0)

assert_dump_time_workspace_pool(snapshot, addr=1000, size=4096, stream=1)
assert_dump_time_workspace_pool(snapshot, addr=10000, size=8192, stream=2)
assert snapshot.device_snapshot.total_allocated == 12288
assert snapshot.device_snapshot.total_activated == 12288


def test_workspace_adapt_warns_and_stops_when_segment_missing(caplog):
data = {
"segments": [_inactive_workspace_segment(2000, 8192, 2)],
"device_traces": [_workspace_triplet(1000, 4096, 1) + _workspace_triplet(2000, 8192, 2)],
}
with caplog.at_level(logging.WARNING, logger="LOAD"):
snapshot = SimulateDeviceSnapshot(data, 0)

assert "Workspace snapshot at addr 1000 (stream 1) not found" in caplog.text
later = _segment_by_addr_stream(snapshot, 2000, 2)
assert later.allocated_size == 0
assert later.active_size == 0
assert later.blocks == []


def test_workspace_adapt_keeps_earlier_group_when_later_group_is_missing(caplog):
data = {
"segments": [
_inactive_workspace_segment(1000, 4096, 1),
_inactive_workspace_segment(10000, 8192, 2),
],
"device_traces": [_workspace_triplet(1000, 4096, 1) + _workspace_triplet(2000, 8192, 2)],
}
with caplog.at_level(logging.WARNING, logger="LOAD"):
snapshot = SimulateDeviceSnapshot(data, 0)

assert_dump_time_workspace_pool(snapshot, addr=1000, size=4096, stream=1)
later = _segment_by_addr_stream(snapshot, 10000, 2)
assert later.allocated_size == 0
assert later.active_size == 0
assert later.blocks == []
assert snapshot.device_snapshot.total_allocated == 4096
assert snapshot.device_snapshot.total_activated == 4096
assert snapshot.device_snapshot.total_reserved == 12288
warnings = [record for record in caplog.records if record.levelno == logging.WARNING]
assert len(warnings) == 1
assert "Workspace snapshot at addr 2000 (stream 2) not found" in caplog.text


def test_workspace_adapt_warns_and_skips_when_size_mismatches(caplog):
data = make_torch_npu_workspace_snapshot(size=4096, segment_size=8192)
with caplog.at_level(logging.WARNING, logger="LOAD"):
snapshot = SimulateDeviceSnapshot(data, 0)

assert "does not match segment total_size" in caplog.text
segment = _segment_by_addr_stream(snapshot, 1000, 1)
assert segment.allocated_size == 0
assert segment.active_size == 0
assert segment.blocks == []
assert snapshot.device_snapshot.total_allocated == 0
assert snapshot.simulated_allocator_context.workspace_flag is True


def test_workspace_adapt_skips_inconsistent_triplet_so_replay_keeps_tolerance(caplog):
data = make_torch_npu_workspace_snapshot()
data["device_traces"][0][2]["size"] = 8192
with caplog.at_level(logging.WARNING):
snapshot = SimulateDeviceSnapshot(data, 0)

segment = _segment_by_addr_stream(snapshot, 1000, 1)
assert segment.allocated_size == 0
assert segment.blocks == []
assert "internally inconsistent" in caplog.text
assert snapshot.replay() is True
assert "workspace scenario tolerance" in caplog.text


def test_workspace_adapt_skips_when_segment_still_has_live_blocks(caplog):
data = {
"segments": [
{
"address": 1000,
"total_size": 4096,
"stream": 1,
"segment_type": "small",
"allocated_size": 1024,
"active_size": 1024,
"device": 0,
"is_expandable": False,
"frames": [],
"blocks": [
{
"size": 1024,
"requested_size": 1024,
"state": "active_allocated",
"address": 1000,
"frames": [],
},
{
"size": 3072,
"requested_size": 3072,
"state": "inactive",
"address": 2048,
"frames": [],
},
],
}
],
"device_traces": [
_workspace_triplet(1000, 4096, 1)
+ [{"action": "alloc", "addr": 1000, "size": 1024, "stream": 1, "frames": []}]
],
}
with caplog.at_level(logging.WARNING, logger="LOAD"):
snapshot = SimulateDeviceSnapshot(data, 0)

segment = _segment_by_addr_stream(snapshot, 1000, 1)
assert len(segment.blocks) == 1
assert segment.blocks[0].size == 1024
assert segment.blocks[0].state == BlockState.ACTIVE_ALLOCATED
assert segment.allocated_size == 1024
assert snapshot.device_snapshot.total_allocated == 1024
assert "still has 1 live block" in caplog.text
Loading