Skip to content

feat: restrict pickle loading to builtins with SafeUnpickler - #102

Merged
codeasier merged 3 commits into
mainfrom
issue-99-safe-unpickler
Aug 19, 2026
Merged

codeasier merged 3 commits into
mainfrom
issue-99-safe-unpickler

Conversation

@codeasier

Copy link
Copy Markdown
Owner

Summary

  • Replace unrestricted pickle.load() in load_pickle_to_dict with SafeUnpickler, allowing only builtins container and scalar types.
  • Import and split share this loader, so custom __reduce__ payloads fail with UnpicklingError / ImportExecutionError / SplitError and do not execute.
  • Docs still say pickle loading is not a sandbox; they now also record the builtins allowlist. PROVENANCE.md records the security-boundary change.

Fixes #99

Test plan

  • Tests added/updated
  • test_load_pickle_to_dict_rejects_unsafe_global_without_running_payload
  • Import and split reject the same payload without executing it
  • Empty cache / expandable / multi-device fixtures still load
  • Corrupt pickle still maps to the existing UnpicklingError wrapper
  • CI green

Checklist

  • Code follows project guidelines
  • Documentation updated (if applicable)
  • No new warnings introduced
  • Changes tested locally

Snapshot Runtime Provenance

  • Snapshot provenance evaluation completed when snapshot runtime files change
    Snapshot provenance decision: updated

Made with Cursor

try:
with open(pickle_file, "rb") as f:
data = pickle.load(f)
data = SafeUnpickler(f).load()

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[minor] SafeUnpickler(f).load() rejections (e.g. Unsafe pickle: global 'os.system' is not allowed) are swallowed by the except Exception: wrapper at line 61 and re-raised as a generic UnpicklingError('Cannot load pickle file: ...') without the cause chain (# noqa: B904, the only such suppression in the repo). No caller regression: import/split map both corrupt-stream and allowlist rejections to the same error surface, and the new tests only assert the exception type. But the specific rejected module.name is lost from logs/forensics. Consider except pickle.UnpicklingError: or raise ... from exc so the allowlist-rejection message is preserved.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Chained the original UnpicklingError with from exc and included {exc} in the wrapper. Allowlist rejections now surface as Cannot load pickle file: <path>: Unsafe pickle: global 'module.name' ..., with __cause__ set. Other non-UnpicklingError failures still use the generic wrapper.

"float",
"bool",
"bytes",
"NoneType",

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] NoneType is a valid GLOBAL lookup target, but CPython pickles None via the NONE opcode, so this entry is defensive/unreachable in normal streams. Harmless to keep (denying it would be moot).

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving NoneType on the allowlist. It matches the issue spec and is only a defensive GLOBAL target; CPython still encodes None with the NONE opcode.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up: hasattr(builtins, "NoneType") is false, so this allowlist entry was dead. super().find_class("builtins", "NoneType") would raise AttributeError and get wrapped as a corrupt-stream error. Removed NoneType. A GLOBAL of builtins.NoneType is now UnsafePickleError. The original PROVENANCE line is unchanged (append-only); the 2026-08-20 entry records the correction.

load_pickle_to_dict(tmp_path / "missing.pkl")


def test_load_pickle_to_dict_rejects_unsafe_global_without_running_payload(tmp_path: Path):

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[minor] This test covers rejection of a non-builtins module global (tests.snapshot.test_file_util.unsafe_pickle_payload), but not the module == 'builtins' disallowed-name branch (e.g. builtins.eval, builtins.open). Suggest parametrizing a few disallowed builtins names to lock that branch.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added find_class coverage for builtins.eval, builtins.exec, and builtins.open. A pickled open object is _io.open, so a dump/load test would miss the module == 'builtins' disallowed-name branch.

Comment thread tests/snapshot/test_file_util.py Outdated
target = tmp_path / "corrupt.pkl"
target.write_bytes(b"\x80\x04not a valid pickle stream")

with pytest.raises(pickle.UnpicklingError, match="Cannot load pickle file"):

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[minor] match="Cannot load pickle file" only pins the generic wrapper prefix, which any wrapped failure would produce — the test would pass even if a valid stream were incorrectly rejected. A stronger assertion (e.g. checking the underlying cause, or dropping match to pin the exact exception) would lock the corrupt-stream behavior.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The corrupt-stream test now asserts the path is in the wrapper, Unsafe pickle is absent, and __cause__ is UnpicklingError.

evidence are unchanged.
- On 2026-08-19, issue #99 replaced unrestricted `pickle.load()` in
`snapshot/util/file_util.py` with `SafeUnpickler`. Unpickling now allows only
`builtins` container and scalar types (`dict`/`list`/`tuple`/`set`/`str`/

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] The allowlist set is duplicated here and in ALLOWED_CLASSES in file_util.py. PROVENANCE is append-only, so this entry is a point-in-time record — acceptable, but a future allowlist change adds a third source of truth alongside the docstring. No action needed beyond awareness.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No change. The PROVENANCE line stays a point-in-time record of the original allowlist; ALLOWED_CLASSES remains the live source of truth.

codeasier and others added 3 commits August 20, 2026 01:29
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@codeasier
codeasier force-pushed the issue-99-safe-unpickler branch from d261c1a to a9f209a Compare August 19, 2026 17:31
@codeasier
codeasier merged commit 3d9cdd2 into main Aug 19, 2026
6 checks passed
@codeasier
codeasier deleted the issue-99-safe-unpickler branch October 3, 2026 19:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE] pickle 加载使用 SafeUnpickler 限制反序列化类型

1 participant