Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ See the [documentation index](docs/README.md) for all English and Chinese guides
| Splitting snapshots | [Splitting Snapshots](docs/en/splitting.md) |
| Agent skills | [Agent Skills](docs/en/skills.md) |
| Database format | [SnapshotDB Schema](docs/en/database.md) |
| Sharded artifact contract (P0; not an exporter) | [Sharded SnapshotDB protocol](docs/en/sharded-snapshotdb.md) |
| Sharded contract and internal replay (P0/P1; no CLI exporter) | [Sharded SnapshotDB protocol](docs/en/sharded-snapshotdb.md) |
| Python API | [SnapshotAnalyzer API](docs/en/snapshot-analyzer-api.md) |
| Result mapping utility | [ResultMapper API](docs/en/result-mapper-api.md) |

Expand Down
2 changes: 1 addition & 1 deletion README_zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ helper 会先路由采集,无需先安装 CLI。分析仍需要 CLI 和 Snapsh
| 拆分快照 | [拆分快照](docs/zh/splitting.md) |
| Agent skill | [Agent Skills](docs/zh/skills.md) |
| 数据库格式 | [SnapshotDB Schema](docs/zh/database.md) |
| 分片产物合同(P0,非导出器) | [分片 SnapshotDB 协议](docs/zh/sharded-snapshotdb.md) |
| 分片合同与内部回放(P0/P1,无 CLI 导出器) | [分片 SnapshotDB 协议](docs/zh/sharded-snapshotdb.md) |
| Python API | [SnapshotAnalyzer API](docs/zh/snapshot-analyzer-api.md) |
| 结果映射工具 | [ResultMapper API](docs/zh/result-mapper-api.md) |

Expand Down
47 changes: 47 additions & 0 deletions docs/en/sharded-snapshotdb.md
Original file line number Diff line number Diff line change
Expand Up @@ -180,6 +180,53 @@ QueryScope("event_range", device_id=0, start_event_id=1, end_event_id=3).validat
assert not inspection.structured_frames
```

## Internal continuous replay (P1)

`core.sharded_replay_service.ShardedReplayService.stage(source, directory,
events_per_slice=..., device=None)` is an **internal producer boundary**, not a
new CLI/API import command. It loads trusted pickle once, constructs one persistent
simulator per selected event-bearing device, and writes windows from last to first
into a new, caller-owned **private staging directory**. `omitted_devices` discloses
all unselected/empty device positions. Existing `import`, `split`, `SnapshotAnalyzer`
and single-DB v1/v2 reading are unchanged.

The returned immutable result identifies **native-v2** slices, not compatibility-v1
artifacts: trace rows use `callstackId` plus a local `callstack` table and retain
native OOM action 8 and workspace events. Sparse/nonzero-start native real IDs are
preserved if nonnegative, unique and chronological; windows count list positions,
not `max(id)+1`. These native slices are **not** accepted by `validate_dataset`.
Compatibility export, a compatible manifest, cache/publication and dataset queries
remain separate work. No `manifest.json`, `readySlices`, or incremental ready
notification is emitted. Only a successful return after all devices finalize and
validate may be handed to later publication. Failure retains private partial
files for their owner; none is advertised as a complete dataset.

`SimulateDeviceSnapshot.replay_until(remaining_events)` undoes until that many
chronological list entries remain. The endpoint is a **position**, not an event ID;
repeated pauses/resumes share the full replay hook/error path. A real event row
records the state **after** its event (before undo). After undoing a window's first
event, active block observations and negative-ID `segment_alloc`/`segment_map`
rows reconstruct the state **before** that first event. Boundary rows do not count
against capacity or real-event peaks.

A device-local registry retains original block objects across writer switches,
not copied hook payloads or an address-only key. Allocation replay resolves every
appearance to the original alloc ID; unknown/preexisting lifetimes retain stable
negative identities. All writers close before per-DB transactional batch backfill
and validation. `freeEventId` stays `free_completed`, never `free_requested`;
boundary `active_pending_free` observations remain state 0. Address reuse and
same-address different-stream blocks remain distinct. Each native slice adds
`pt_snap_block_reference(blockId, stream, allocCallstack, freeCallstack)` so finalized
out-of-slice lifecycle references do not discard their text stack sources. NULL
means no observed source event; empty text means the observed event had no frames.
Legacy slice-local query JOINs do not automatically consume this internal table;
this is not a cross-slice query engine or structured-frame extension.

Pickle may still load entirely. Sharding bounds **real events per DB**, not block
rows, total staged bytes, registry size, or peak RSS. Synthetic regressions and
reviewed expandable/multi-device fixtures cover replay equivalence and resource
failure paths; no live upstream GUI or performance acceptance is claimed.

## Evidence and limits

Interoperability facts were read from fixed upstream
Expand Down
36 changes: 36 additions & 0 deletions docs/zh/sharded-snapshotdb.md
Original file line number Diff line number Diff line change
Expand Up @@ -148,6 +148,42 @@ QueryScope("event_range", device_id=0, start_event_id=1, end_event_id=3).validat
assert not inspection.structured_frames
```

## 内部持续回放(P1)

`core.sharded_replay_service.ShardedReplayService.stage(source, directory,
events_per_slice=..., device=None)` 是**内部生产边界**,不是新 CLI/API 导入命令。
它只加载一次可信 pickle,每个所选非空设备只创建一个持续模拟器,从末片向前写入
新的、调用者所有的**私有 staging 目录**。`omitted_devices` 披露全部未选/空设备位置。
现有 `import`、`split`、`SnapshotAnalyzer` 与单库 v1/v2 读取保持不变。

返回的不可变结果标识 **native-v2** 分片,不是兼容 v1 产物:trace 使用
`callstackId` 和片内 `callstack` 表,保留原生 OOM action 8 与 workspace 事件。
原生真实 ID 可以稀疏或非零起点,但必须非负、唯一且按时间递增;窗口按列表位置计数,
不是 `max(id)+1`。这些原生片**不能**通过 `validate_dataset`。兼容导出、兼容 manifest、
缓存/发布及数据集查询属于后续独立工作。不生成 `manifest.json`、`readySlices` 或增量
ready 通知;仅所有设备完成回填及校验后的成功返回结果可交给后续发布层。
失败保留调用者所有的私有部分文件,不将其宣称为完整数据集。

`SimulateDeviceSnapshot.replay_until(remaining_events)` 倒序回放到剩余指定数量的
时间顺序列表事件。端点是**位置**而非 event ID;重复暂停/继续与完整 replay 共用
hooks 和错误路径。真实事件行记录事件**之后**的状态(undo 之前)。undo 窗口首事件后,
活跃 block 观察和负 ID `segment_alloc`/`segment_map` 行重建首事件**之前**的状态;
边界行不计入容量或真实事件峰值。

设备内 registry 跨 writer 保留原始 block 对象,不用 hook 副本或仅地址作为身份。
遇到真实 alloc 后,每次出现都最终指向原 alloc ID;未知/预存生命周期保持稳定负身份。
所有 writer 关闭后才逐 DB 事务批量回填并校验。`freeEventId` 始终是 `free_completed`,
不是 `free_requested`;边界 `active_pending_free` 观察保留 state 0。同地址复用及不同
stream 的同地址块不会合并。每个原生片加入
`pt_snap_block_reference(blockId, stream, allocCallstack, freeCallstack)`,保留最终化的
片外生命周期引用的文本调用栈来源。NULL 表示未观察到来源事件;空文本表示已观察事件
没有 frames。旧有片内查询 JOIN 不自动消费这个内部表;它不是跨片查询引擎或结构化
frame 扩展。

pickle 仍可能整体加载。分库只限制**单库真实事件数**,不限制 block 行数、总 staging
字节、registry 大小或峰值 RSS。合成回归及已审 expandable/多设备 fixture 覆盖回放
等价和资源失败路径;不宣称真实上游 GUI 或性能验收。

## 来源证据与限制

接口事实已从固定版本真实源码读取:
Expand Down
57 changes: 57 additions & 0 deletions src/pt_snap_cli/core/sharded_replay_service.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
"""Internal P1 shared producer boundary, not a CLI import/export facade."""

from __future__ import annotations

from dataclasses import dataclass
from pathlib import Path
from typing import cast

from ..snapshot.representation import load_snapshot_representation
from ..snapshot.tools.adaptors.sharded_replay import ReplaySlice, build_shards


@dataclass(frozen=True)
class ShardedReplayResult:
directory: Path
slices: tuple[ReplaySlice, ...]
omitted_devices: tuple[int, ...]
layout: str = "native-v2"


class ShardedReplayService:
def stage(
self,
source: Path | str,
directory: Path | str,
*,
events_per_slice: int,
device: int | None = None,
) -> ShardedReplayResult:
"""Load once and finalize native shards in a NEW private staging directory.

The caller owns partial files on failure. No manifest, ready slice, focus,
cache reuse or final publication is exposed. Only a successful result can
be handed to a subsequent exporter/publication service. Pickle is trusted
executable input; this does not implement streaming or an RSS bound.
"""
if type(events_per_slice) is not int or events_per_slice <= 0:
raise ValueError("events_per_slice must be a positive integer")
if device is not None and (type(device) is not int or device < 0):
raise ValueError("device must be a nonnegative integer")
target = Path(directory).absolute()
if target.resolve() != target or target.is_symlink():
raise ValueError("Staging directory must be canonical and non-symlink")
if target.exists():
raise FileExistsError(target)
data = load_snapshot_representation(Path(source))
traces = cast(list[list[object]], data.get("device_traces", []))
available = tuple(index for index, entries in enumerate(traces) if entries)
if not available or (device is not None and device not in available):
raise ValueError("Requested device is absent or has no trace events")
selected = available if device is None else (device,)
# Exclusive allocation: never replace or adopt an existing directory.
target.mkdir()
slices = build_shards(data, target, events_per_slice, selected)
return ShardedReplayResult(
target, slices, tuple(index for index in range(len(traces)) if index not in selected)
)
25 changes: 25 additions & 0 deletions src/pt_snap_cli/snapshot/PROVENANCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -228,3 +228,28 @@ discarded side-effect return values explicit. Database/slice adaptor consumers
annotate the existing per-device trace-list and segment-list format assumptions.
No serialization, validation, replay, or database layout behavior changes.
Audited upstream source mappings and license terms remain unchanged.

## 2026-10-05 — Continuous native shard replay (issue #199)

Interface facts were independently read through real GitHub GETs at
Ascend/msinsight revision `101f65b877a267ffd5f66ea3834706057ba243e5`:
[resumable replay](https://api.github.com/repos/Ascend/msinsight/git/blobs/1956490d46ccf11309d10853ea76175d0713934e)
and [boundary/registry interfaces](https://api.github.com/repos/Ascend/msinsight/git/blobs/95612509ef990ac6b7034919dae4938cc6acf3ad).
The fixed tree confirms both blob identities. These sources carry Copyright (c)
2026 Huawei Technologies Co.,Ltd. and Mulan PSL v2; the docs' CC BY 4.0 license
is separate. No upstream implementation is copied into the package.

Local original implementation changes `simulate/simulate.py` to reuse the existing
full replay loop for position-based pauses (deliberately not an ID threshold),
adds `tools/adaptors/sharded_replay.py` for native-v2 multi-writer staging,
original-object lifecycle tracking, negative left-boundary rows, transactional
batch ID/free_completed/stack-source backfill and final validation, and adds the
shared `core/sharded_replay_service.py` producer boundary. Unknown allocations
keep device-local negative identities without upstream compaction. All writers
close before finalization; no incremental readiness, compatibility manifest,
cache/publication, CLI changes, or structured-frame extension is introduced.
`tools/adaptors/snapshot2db.py` and `database/snapshot_db.py` close connections on
constructor/commit failure while preserving the original single-DB data contract.
Synthetic failure/equivalence tests and bilingual docs are locally authored;
reviewed fixtures are unchanged. Existing source mappings and MIT relicensing
records are retained. GUI interoperability and streaming/RSS claims are not made.
25 changes: 21 additions & 4 deletions src/pt_snap_cli/snapshot/simulate/simulate.py
Original file line number Diff line number Diff line change
Expand Up @@ -154,14 +154,27 @@ def unregister_allocator_hooker(self, hooker_id: int):
self.simulated_allocator.unregister_hooker(hooker_id)

def replay(self) -> bool:
"""Undo the entire remaining trace through the resumable replay path."""
return self.replay_until(0)

def replay_until(self, remaining_events: int) -> bool:
"""Undo until exactly ``remaining_events`` chronological entries remain.

The endpoint is a list position, not an event ID (native IDs may be sparse).
Undoing positions [start, end] leaves the state immediately BEFORE start.
Pre hooks observe the state AFTER the event; post hooks observe BEFORE it.
A pre/allocator failure leaves the event pending, while a post-hook failure
occurs after it was consumed, exactly as for full replay.
"""
开始仿真回放内存事件
"""
if type(remaining_events) is not int or not (
0 <= remaining_events <= len(self.device_snapshot.trace_entries)
):
raise ValueError("remaining_events must be an integer within the remaining trace")
# 倒序遍历
total_size = len(self.device_snapshot.trace_entries)
self._replay_logger.info(f"Replaying {total_size} entries in snapshot...")
progress_update_point = [0.25, 0.5, 0.75]
while self.device_snapshot.trace_entries:
while len(self.device_snapshot.trace_entries) > remaining_events:
for hooker in self.hookers.values():
if hooker and not hooker.pre_undo_event(
self.device_snapshot.trace_entries[-1], self.device_snapshot
Expand Down Expand Up @@ -191,5 +204,9 @@ def replay(self) -> bool:
"An interruption occurred during the replay of the single event post hook."
)
return False
self._replay_logger.info("All events have been successfully replayed.")
self._replay_logger.info(
"All events have been successfully replayed."
if not remaining_events
else f"Replay paused with {remaining_events} entries remaining."
)
return True
19 changes: 12 additions & 7 deletions src/pt_snap_cli/snapshot/tools/adaptors/database/snapshot_db.py
Original file line number Diff line number Diff line change
Expand Up @@ -64,13 +64,18 @@ class SnapshotDb(SqliteDB):
CALLSTACK_TABLE_NAME = "callstack"

def __init__(self, path: str):
super().__init__(path, auto_create=True, with_dictionary_table=True)
# Import builds a replaceable database, so favor write throughput over crash recovery.
self.conn.execute("PRAGMA journal_mode = MEMORY")
self.conn.execute("PRAGMA synchronous = OFF")
self.conn.execute("PRAGMA cache_size = -65536")
# 清理旧版本表格
self._clear_old_tables()
try:
super().__init__(path, auto_create=True, with_dictionary_table=True)
# Import builds a replaceable database, so favor write throughput over crash recovery.
self.conn.execute("PRAGMA journal_mode = MEMORY")
self.conn.execute("PRAGMA synchronous = OFF")
self.conn.execute("PRAGMA cache_size = -65536")
# 清理旧版本表格
self._clear_old_tables()
except BaseException:
if hasattr(self, "conn"):
self.conn.close()
raise

def create_trace_entry_table(self, device: int = 0):
self.create_table(
Expand Down
Loading