Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,8 @@ pip install -e .
To start from a raw PyTorch memory snapshot, import the pickle into a SnapshotDB with pt-snap's built-in snapshot support, then list the available query templates:

> **Security warning:** Import only pickle snapshots from a trusted source. Pickle
> deserialization can execute arbitrary code. `pt-snap import` is not a sandbox.
> deserialization can execute arbitrary code. The loader rejects non-`builtins`
> global objects, but `pt-snap import` is not a sandbox.

```bash
pt-snap import snapshot.pkl
Expand Down
2 changes: 1 addition & 1 deletion README_zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ pip install -e .
## 快速开始

> **安全警告:** 只导入来自可信来源的 pickle 快照。反序列化 pickle 可能执行任意代码;
> `pt-snap import` 不是沙箱。
> 实现会拒绝非 builtins 全局对象,但 `pt-snap import` 不是沙箱。

```bash
# 从原始 pickle 导入;相同内容和配置会自动复用已有 DB
Expand Down
3 changes: 2 additions & 1 deletion docs/en/database.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,8 @@ SnapshotDB is the SQLite database format for persisting PyTorch memory profiling
`pt-snap` analyzes SQLite SnapshotDB files. If you start with a raw PyTorch memory snapshot pickle, import the snapshot with the built-in backend:

> **Security warning:** Import trusted pickle files only. Pickle deserialization
> can execute arbitrary code. `pt-snap import` is not a sandbox.
> can execute arbitrary code. The loader rejects non-`builtins` global objects,
> but `pt-snap import` is not a sandbox.

```bash
pt-snap import snapshot.pkl
Expand Down
3 changes: 2 additions & 1 deletion docs/en/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,8 @@ pip install -e .
If you have a raw `.pkl` memory snapshot, import it first with the built-in backend:

> **Security warning:** Use trusted pickle input only. Deserializing a pickle can
> execute arbitrary code. `pt-snap import` is not a sandbox.
> execute arbitrary code. The loader rejects non-`builtins` global objects, but
> `pt-snap import` is not a sandbox.

```bash
pt-snap import snapshot.pkl
Expand Down
4 changes: 2 additions & 2 deletions docs/en/splitting.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@ event range. It does not truncate trace arrays, read focus, update focus, or cre
a SnapshotDB. Import a produced pickle separately when you need SQLite queries.

> **Security warning:** Use trusted pickle input only. Deserializing the source
> snapshot and any pickle slice can execute arbitrary code. `pt-snap split` is
> not a sandbox.
> snapshot and any pickle slice can execute arbitrary code. The loader rejects
> non-`builtins` global objects, but `pt-snap split` is not a sandbox.

## Command Contract

Expand Down
2 changes: 1 addition & 1 deletion docs/zh/database.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ SnapshotDB 是内存快照数据的 SQLite 数据库存储格式,用于持久
如果输入是 PyTorch 原始 `.pkl` 内存快照,可以直接导入:

> **安全警告:** 只能导入可信的 pickle 文件。反序列化 pickle 可能执行任意代码;
> `pt-snap import` 不是沙箱。
> 实现会拒绝非 builtins 全局对象,但 `pt-snap import` 不是沙箱。

```bash
pt-snap import snapshot.pkl
Expand Down
2 changes: 1 addition & 1 deletion docs/zh/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ pip install -e .
如果你手上是原始 `.pkl` 内存快照,请先使用内建后端导入:

> **安全警告:** 只能使用可信的 pickle 输入。反序列化 pickle 可能执行任意代码;
> `pt-snap import` 不是沙箱。
> 实现会拒绝非 builtins 全局对象,但 `pt-snap import` 不是沙箱。

```bash
pt-snap import snapshot.pkl
Expand Down
2 changes: 1 addition & 1 deletion docs/zh/splitting.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
focus,也不会生成 SnapshotDB。如需使用 SQLite 查询,请单独导入生成的 pickle 切片。

> **安全警告:** 只能使用可信的 pickle 输入。反序列化源快照或任意 pickle 切片都可能
> 执行任意代码;`pt-snap split` 不是沙箱。
> 执行任意代码;实现会拒绝非 builtins 全局对象,但 `pt-snap split` 不是沙箱。

## 命令契约

Expand Down
3 changes: 3 additions & 0 deletions src/pt_snap_cli/core/snapshot_import_backend.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
from pathlib import Path

from pt_snap_cli.core.errors import ImportExecutionError, ImportToolMissingError
from pt_snap_cli.snapshot.representation import UnsafePickleError


class SnapshotImportBackend:
Expand All @@ -34,6 +35,8 @@ def dump_to_db(
dump_dir=str(tmp_dir),
device=device,
)
except UnsafePickleError as exc:
raise ImportExecutionError(f"Snapshot pickle rejected: {exc}") from exc
except (OSError, sqlite3.DatabaseError, pickle.UnpicklingError, ValueError) as exc:
raise ImportExecutionError(f"Snapshot import backend failed: {exc}") from exc

Expand Down
3 changes: 3 additions & 0 deletions src/pt_snap_cli/core/split_service.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
from pt_snap_cli.core.errors import SplitError, SplitPhase
from pt_snap_cli.core.models import SplitFormat, SplitOptions, SplitResult
from pt_snap_cli.snapshot.representation import (
UnsafePickleError,
load_snapshot_representation,
replay_snapshot,
save_snapshot_representation,
Expand All @@ -40,6 +41,8 @@ def split(self, options: SplitOptions) -> SplitResult:

try:
representation = load_snapshot_representation(source, "pickle")
except UnsafePickleError as exc:
self._fail("load/engine", source, f"unsafe pickle: {exc}", exc)
except Exception as exc:
self._fail("load/engine", source, str(exc), exc)

Expand Down
20 changes: 20 additions & 0 deletions src/pt_snap_cli/snapshot/PROVENANCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,26 @@ The upstream standalone split frontend is deliberately excluded, not deferred:
read `device_free`, and mapped `"oom"` to action value `8` so import persists the
event as an integer. Replay still skips OOM. This is a local first-party bugfix;
audited upstream mappings and licensing evidence are unchanged.
- On 2026-08-19, issue #99 replaced unrestricted `pickle.load()` in
`snapshot/util/file_util.py` with `SafeUnpickler`. Unpickling now allows only
`builtins` container and scalar types (`dict`/`list`/`tuple`/`set`/`str`/

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] The allowlist set is duplicated here and in ALLOWED_CLASSES in file_util.py. PROVENANCE is append-only, so this entry is a point-in-time record — acceptable, but a future allowlist change adds a third source of truth alongside the docstring. No action needed beyond awareness.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No change. The PROVENANCE line stays a point-in-time record of the original allowlist; ALLOWED_CLASSES remains the live source of truth.

`int`/`float`/`bool`/`bytes`/`NoneType`); any other `module.name` raises
`UnpicklingError`. Import and split share this loader via
`representation.load_pickle_representation`. The change tightens the
deserialization allowlist; pickle loading remains not a sandbox. Corrupt
streams still map to the established generic `UnpicklingError` wrapper.
- On 2026-08-19, PR #102 follow-up kept that wrapper prefix and path, and chained
the original `UnpicklingError` so allowlist rejections keep the rejected
`module.name` in the message and `__cause__`.
- On 2026-08-20, PR #102 follow-up removed `NoneType` from `ALLOWED_CLASSES`.
`builtins.NoneType` is not an importable builtin (`hasattr(builtins, "NoneType")`
is false); `None` is encoded by the `NONE` opcode. A GLOBAL of
`builtins.NoneType` is now `UnsafePickleError` rather than an `AttributeError`
swallowed as a corrupt-stream wrapper. `UnsafePickleError` subclasses
`pickle.UnpicklingError`. Import maps it to `Snapshot pickle rejected:` and
split maps it to `unsafe pickle:` so allowlist rejection is distinct from
corrupt input. `dump()` re-raises `UnsafePickleError` instead of returning
false.

## Migration toolchain

Expand Down
1 change: 1 addition & 0 deletions src/pt_snap_cli/snapshot/representation.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
from typing import Any, Literal, cast

from .simulate import AllocatorHooker, SimulateDeviceSnapshot, SimulateHooker
from .util.file_util import UnsafePickleError as UnsafePickleError
from .util.file_util import load_pickle_to_dict, save_dict_to_pickle

SnapshotFormat = Literal["pickle", "json"]
Expand Down
4 changes: 3 additions & 1 deletion src/pt_snap_cli/snapshot/tools/adaptors/snapshot2db.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
from pathlib import Path

from ...base import Block, BlockState, DeviceSnapshot, TraceEntry
from ...representation import load_snapshot_representation, replay_snapshot
from ...representation import UnsafePickleError, load_snapshot_representation, replay_snapshot
from ...simulate import AllocatorHooker, SimulateHooker
from ...util.logger import get_logger, set_global_log_file
from .database import SnapshotDb, block2record, event2record
Expand Down Expand Up @@ -138,6 +138,8 @@ def close(self, *, commit: bool = True):
def dump(pickle_file: str, dump_file: str, device=None) -> bool:
try:
data = load_snapshot_representation(Path(pickle_file))
except UnsafePickleError:
raise
except Exception as e:
dump_logger.error(f"Failed to load pickle file: {e}")
return False
Expand Down
39 changes: 37 additions & 2 deletions src/pt_snap_cli/snapshot/util/file_util.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,36 @@
from typing import Any


class UnsafePickleError(pickle.UnpicklingError):
"""Raised when unpickling requests a global outside the builtins allowlist."""


class SafeUnpickler(pickle.Unpickler):
"""Allow only builtins container and scalar types during unpickling."""

ALLOWED_CLASSES = {
"builtins": {
"dict",
"list",
"tuple",
"set",
"str",
"int",
"float",
"bool",
"bytes",
},
}

def find_class(self, module: str, name: str):
if module in self.ALLOWED_CLASSES and name in self.ALLOWED_CLASSES[module]:
return super().find_class(module, name)
raise UnsafePickleError(
f"Unsafe pickle: global '{module}.{name}' is not allowed. "
"Only basic Python types are permitted for security."
)


def load_pickle_to_dict(pickle_file: Path) -> dict:
"""
从指定路径加载 pickle 文件,并确保其内容为 dict 类型。
Expand All @@ -23,14 +53,19 @@ def load_pickle_to_dict(pickle_file: Path) -> dict:
Raises:
FileNotFoundError: 文件不存在
ValueError: 文件内容不是 dict 类型
pickle.UnpicklingError: 反序列化失败(如文件损坏或非 pickle 格式)
pickle.UnpicklingError: 反序列化失败(文件损坏或非 pickle 格式)
UnsafePickleError: 请求了 allowlist 之外的全局对象
"""
if not pickle_file.is_file():
raise FileNotFoundError(f"Cannot found pickle file: {pickle_file}")

try:
with open(pickle_file, "rb") as f:
data = pickle.load(f)
data = SafeUnpickler(f).load()

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[minor] SafeUnpickler(f).load() rejections (e.g. Unsafe pickle: global 'os.system' is not allowed) are swallowed by the except Exception: wrapper at line 61 and re-raised as a generic UnpicklingError('Cannot load pickle file: ...') without the cause chain (# noqa: B904, the only such suppression in the repo). No caller regression: import/split map both corrupt-stream and allowlist rejections to the same error surface, and the new tests only assert the exception type. But the specific rejected module.name is lost from logs/forensics. Consider except pickle.UnpicklingError: or raise ... from exc so the allowlist-rejection message is preserved.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Chained the original UnpicklingError with from exc and included {exc} in the wrapper. Allowlist rejections now surface as Cannot load pickle file: <path>: Unsafe pickle: global 'module.name' ..., with __cause__ set. Other non-UnpicklingError failures still use the generic wrapper.

except UnsafePickleError:
raise
except pickle.UnpicklingError as exc:
raise pickle.UnpicklingError(f"Cannot load pickle file: {pickle_file}: {exc}") from exc
except Exception:
# Preserve the established exception context and corrupt-pickle behavior.
raise pickle.UnpicklingError(f"Cannot load pickle file: {pickle_file}") # noqa: B904
Expand Down
8 changes: 8 additions & 0 deletions tests/core/test_import_errors.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,3 +20,11 @@ def test_snapshot_file_invalid_error_is_pt_snap_core() -> None:

def test_import_execution_error_is_pt_snap_core() -> None:
assert issubclass(ImportExecutionError, PtSnapCoreError)


def test_unsafe_pickle_error_is_unpickling_error() -> None:
from pickle import UnpicklingError

from pt_snap_cli.snapshot.representation import UnsafePickleError

assert issubclass(UnsafePickleError, UnpicklingError)
13 changes: 13 additions & 0 deletions tests/core/test_import_service.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
SnapshotFileInvalidError,
)
from pt_snap_cli.core.models import ImportOptions
from tests.snapshot.test_file_util import UNSAFE_PICKLE_PAYLOAD, write_unsafe_reduce_pickle

FIXTURE_DIR = Path(__file__).parents[1] / "fixtures" / "snapshots"
EMPTY_CACHE_SNAPSHOT = FIXTURE_DIR / "snapshot_with_empty_cache.pkl"
Expand Down Expand Up @@ -122,6 +123,18 @@ def test_import_snapshot_file_invalid_corrupt_pickle(tmp_path: Path) -> None:
assert not (tmp_path / "corrupt.pkl.db").exists()


def test_import_rejects_pickle_with_non_allowlisted_global(tmp_path: Path) -> None:
import_service_cls = _import_service_type()
service = import_service_cls()
snapshot_file = write_unsafe_reduce_pickle(tmp_path / "evil.pkl")

with pytest.raises(ImportExecutionError, match=r"Snapshot pickle rejected:.*Unsafe pickle"):
service.import_snapshot(ImportOptions(snapshot_file=snapshot_file))

assert UNSAFE_PICKLE_PAYLOAD["executed"] is False
assert not (tmp_path / "evil.pkl.db").exists()


def test_failed_reimport_preserves_existing_database(tmp_path: Path) -> None:
import_service_cls = _import_service_type()
service = import_service_cls()
Expand Down
13 changes: 13 additions & 0 deletions tests/core/test_split_service.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@
from pt_snap_cli.snapshot.tools.slice_dump.hooker import SliceDumpHooker
from tests.snapshot.golden_observations import ACTION_VALUE_MAP
from tests.snapshot.helpers import FIXTURE_DIR, assert_valid_snapshot
from tests.snapshot.test_file_util import UNSAFE_PICKLE_PAYLOAD, write_unsafe_reduce_pickle

MULTI_DEVICE = FIXTURE_DIR / "snapshot_with_multi_devices.pkl"

Expand Down Expand Up @@ -383,6 +384,18 @@ def test_malformed_pickle_is_load_engine_phase(tmp_path: Path) -> None:
assert not (tmp_path / "split").exists()


def test_unsafe_pickle_global_is_load_engine_phase(tmp_path: Path) -> None:
source = write_unsafe_reduce_pickle(tmp_path / "evil.pkl")

with pytest.raises(SplitError, match="unsafe pickle") as caught:
SplitService().split(_options(tmp_path, snapshot_file=source))
assert caught.value.phase == "load/engine"
assert "Unsafe pickle" in caught.value.detail
assert str(source) in str(caught.value)
assert UNSAFE_PICKLE_PAYLOAD["executed"] is False
assert not (tmp_path / "split").exists()


def test_no_nonempty_device_is_device_phase(tmp_path: Path) -> None:
source = tmp_path / "empty.pkl"
source.write_bytes(pickle.dumps({"segments": [], "device_traces": [[], []]}))
Expand Down
69 changes: 69 additions & 0 deletions tests/snapshot/test_file_util.py
Original file line number Diff line number Diff line change
@@ -1,14 +1,36 @@
import io
import pickle
from pathlib import Path

import pytest

from pt_snap_cli.snapshot.util.file_util import (
SafeUnpickler,
UnsafePickleError,
check_dir_valid,
check_file_valid,
load_pickle_to_dict,
save_dict_to_pickle,
)
from tests.snapshot.helpers import FIXTURE_DIR

UNSAFE_PICKLE_PAYLOAD = {"executed": False}


def unsafe_pickle_payload() -> dict:
UNSAFE_PICKLE_PAYLOAD["executed"] = True
return {"pwned": True}


class UnsafeReducePayload:
def __reduce__(self) -> tuple[object, tuple[()]]:
return (unsafe_pickle_payload, ())


def write_unsafe_reduce_pickle(path: Path) -> Path:
UNSAFE_PICKLE_PAYLOAD["executed"] = False
path.write_bytes(pickle.dumps({"payload": UnsafeReducePayload()}))
return path


def test_save_and_load_pickle_round_trip(tmp_path: Path):
Expand All @@ -35,6 +57,53 @@ def test_load_pickle_to_dict_rejects_missing_file(tmp_path: Path):
load_pickle_to_dict(tmp_path / "missing.pkl")


def test_load_pickle_to_dict_rejects_unsafe_global_without_running_payload(tmp_path: Path):

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[minor] This test covers rejection of a non-builtins module global (tests.snapshot.test_file_util.unsafe_pickle_payload), but not the module == 'builtins' disallowed-name branch (e.g. builtins.eval, builtins.open). Suggest parametrizing a few disallowed builtins names to lock that branch.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added find_class coverage for builtins.eval, builtins.exec, and builtins.open. A pickled open object is _io.open, so a dump/load test would miss the module == 'builtins' disallowed-name branch.

target = write_unsafe_reduce_pickle(tmp_path / "evil.pkl")

with pytest.raises(
UnsafePickleError,
match=r"Unsafe pickle: global 'tests\.snapshot\.test_file_util\.unsafe_pickle_payload'",
):
load_pickle_to_dict(target)

assert UNSAFE_PICKLE_PAYLOAD["executed"] is False


@pytest.mark.parametrize("name", ("eval", "exec", "open", "NoneType"))
def test_safe_unpickler_rejects_disallowed_builtin_names(name: str):
unpickler = SafeUnpickler(io.BytesIO(b""))

with pytest.raises(UnsafePickleError, match=rf"Unsafe pickle: global 'builtins\.{name}'"):
unpickler.find_class("builtins", name)


def test_load_pickle_to_dict_rejects_corrupt_stream(tmp_path: Path):
target = tmp_path / "corrupt.pkl"
target.write_bytes(b"\x80\x04not a valid pickle stream")

with pytest.raises(pickle.UnpicklingError, match="Cannot load pickle file") as caught:
load_pickle_to_dict(target)

message = str(caught.value)
assert str(target) in message
assert "Unsafe pickle" not in message
assert not isinstance(caught.value, UnsafePickleError)
assert isinstance(caught.value.__cause__, pickle.UnpicklingError)


@pytest.mark.parametrize(
"fixture_name",
(
"snapshot_with_empty_cache.pkl",
"snapshot_expandable.pkl",
"snapshot_with_multi_devices.pkl",
),
)
def test_load_pickle_to_dict_accepts_fixture_snapshots(fixture_name: str):
data = load_pickle_to_dict(FIXTURE_DIR / fixture_name)
assert isinstance(data, dict)


def test_save_dict_to_pickle_rejects_non_dict(tmp_path: Path):
with pytest.raises(TypeError):
save_dict_to_pickle([1, 2, 3], tmp_path / "bad.pkl")
Expand Down
Loading
Loading