Skip to content

feat: vendor MemSnapDump dump2db backend into pt-snap-cli - #67

Merged
codeasier merged 3 commits into
mainfrom
feat/memsnapdump-vendoring
Jun 22, 2026
Merged

codeasier merged 3 commits into
mainfrom
feat/memsnapdump-vendoring

Conversation

@codeasier

@codeasier codeasier commented Jun 21, 2026 •

Copy link
Copy Markdown
Owner

📝 Description

把 pt-snap import 从 外部 optional dependency 升级为 内建核心能力:将 MemSnapDump 的 dump2db 最小闭包 vendoring 进仓,不再依赖运行时外部安装 memsnapdump。

与 PR #66 的关系:
PR #66 是原型验证 PR,证明 pt-snap import 的链路 / 接口 / schema 兼容性可行,但保留外部 git 依赖。本 PR 是 vendoring 正式版,直接落地"pt-snap import 是核心能力,不应依赖运行时外部包"的目标函数。
建议合并本 PR;不要合并 PR #66。

实现要点:

  • vendor 入口仍是 memsnapdump.tools.adaptors.snapshot2db.run_dump_to_db(不是 issue 提到的 tools.dump2db,后者只有 main())。
  • 23 个上游 Python 源码文件(21 个闭包主体 + 2 个 __init__.py)按闭包定义原样搬入 src/pt_snap_cli/vendor/memsnapdump/,保留上游模块结构(方便未来 diff / 同步)。不携带任何 __pycache__/*.pyc 字节码缓存。
  • 新增语义化包装层 src/pt_snap_cli/core/snapshot_import_backend.py,作为唯一与 vendor 对接的边界。
  • ImportService 改走 vendored backend,公开接口形状完全冻结。
  • ImportToolMissingError 语义改为"vendor backend 不可用 / 打包异常",不再是"用户漏装 memsnapdump"。
  • 接口层(ImportOptions / ImportResult / 3 个错误类型 / pt-snap import CLI 形状)与 PR feat: add pt-snap import to ingest PyTorch memory snapshot pickles via MemSnapDump #66 完全一致。

依赖与 CI:

  • 删除 pyproject.toml 中的 memsnapdump @ git+...@master 可选依赖。
  • .github/workflows/test.yml 改为只装 .[dev],不再安装 memsnapdump 外部包 —— 这保证 vendor 自包含,任何 packaging 漏洞在 CI 都会立即暴露。
  • black / ruff 加 extend-exclude = ["src/pt_snap_cli/vendor"],避免上游代码被反复风格化重写。

🔗 Related Issue

Fixes #65

🧪 Testing

  • Tests added/updated — 22 个 import 相关测试(models / errors / ImportService / TestImportCommand)
  • Manual testing completed — 真实 fixture 全链路:pt-snap import sample.pkl → 生成 11MB db → pt-snap focus → pt-snap query --template-use block 返回 3219 行

本地验证(全绿):

  • pytest -q: 314 passed
  • ruff check .: All checks passed
  • black --check .: 58 files unchanged
  • basedpyright: 0 errors, 0 warnings, 0 notes
  • e2e smoke:
    • pt-snap import sample.pkl → 生成 <name>.db + focus 自动写入
    • pt-snap focus 显示新 DB
    • pt-snap query --template-use block 返回 3219 个结果

✅ Checklist

  • Code follows project guidelines
  • Documentation updated (README / docs/en|zh/quickstart / docs/en/database)
  • No new warnings introduced(ruff / black / basedpyright 全绿)
  • Changes tested locally

📋 Additional Notes

与 issue #65 验收标准对照:

  • 把 .pkl / .pickle 转成 pt-snap-cli 可分析的 SQLite .db(本 PR 通过 vendoring 实现,不需要可选 extra)
  • 默认将产物写入项目级 .pt-snap/focus.json
  • 支持 --no-focus / --output-dir / --device
  • 不实现 split / clipping
  • 不引入 API / MCP 镜像
  • 不引入 subprocess fallback
  • 使用真实 .pkl fixture 覆盖主路径测试(snapshot_with_empty_cache.pkl + snapshot_with_multi_devices.pkl)

PR 描述差异(对比 PR #66):

vendor 同步策略:vendor 子树保留上游结构与文件名,后续同步上游 commit / tag 时可直接 diff,不再吃版本漂移或分支状态变化。

out of scope(与 PR #66 一致):pt-snap split 时序切片、API / MCP 镜像、subprocess fallback。

pt-snap import is now a built-in core capability: the dump2db backend
(used to convert raw PyTorch memory snapshot pickles into pt-snap-cli
SQLite databases) is vendored under src/pt_snap_cli/vendor/memsnapdump/.

Interface layer is frozen: ImportService, ImportOptions, ImportResult,
the three error types, and the pt-snap import CLI shape are unchanged
from the prototype PR. What changes is where the backend lives.

Architecture:
- src/pt_snap_cli/vendor/memsnapdump/ holds the minimal closure required
  to execute run_dump_to_db, with the upstream module structure
  preserved so future syncs remain diff-friendly.
- src/pt_snap_cli/core/snapshot_import_backend.py is the semantic
  adapter over the vendored backend, isolating the rest of the code
  from the vendor namespace.
- ImportService now calls SnapshotImportBackend instead of importing
  from the external memsnapdump package.
- ImportToolMissingError is repurposed: it now means the vendored
  backend could not be loaded (a packaging problem), not that the
  user forgot to install memsnapdump.

Dependency and CI:
- pyproject.toml: the optional 'memsnapdump' git extra is removed.
  No runtime external package is required for pt-snap import.
- .github/workflows/test.yml: the test job installs only .[dev] and
  must pass without any extra memsnapdump installation, ensuring
  vendoring is genuinely self-contained.
- black and ruff exclude src/pt_snap_cli/vendor/ so the upstream
  code is not reformatted on every lint pass.

Tests:
- The vendored backend is exercised against real .pkl fixtures under
  tests/fixtures/snapshots/, including a multi-device snapshot, so
  the closure's coverage of replay/allocator paths is verified.
- CLI tests (TestImportCommand) cover the success path, --no-focus,
  missing file, bad suffix, and missing-tool display.
- Service tests cover input validation, focus writing, and the
  per-error exception paths.

Out of scope (unchanged from the prototype PR): split/clipping,
API/MCP mirror, subprocess fallback. These belong to separate follow-ups.

Refs #65
@codeasier
codeasier force-pushed the feat/memsnapdump-vendoring branch from 41b6bb8 to 4fe5555 Compare June 21, 2026 17:28
Comment thread src/pt_snap_cli/core/import_service.py Outdated
) -> None:
self._focus_service = focus_service
self._backend = backend or SnapshotImportBackend()
self._backend_available = True

@codeasier codeasier Jun 21, 2026 •

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Blocker 1: _backend_available 是死代码

__init__ 之后 _backend_available 永远是 True,且无任何代码路径会把它设回 False。_run_dump_to_db 在 __init__ 也立即被绑定,不可能为 None。但 _ensure_backend_available() 还是检查了它们 — 这两个分支在生产路径里永远走不到。

而测试 test_import_raises_on_missing_tool 通过 service._backend_available = False 强行触发,说明这个分支只是为了测试而存在。

建议二选一:

  1. 彻底删除 _backend_available / _ensure_backend_available() / assert self._run_dump_to_db is not None,把 vendor 不可用的情况全部交给 SnapshotImportBackend 的 ImportError catch(见 snapshot_import_backend.py),语义统一。
  2. 真正实现 backend 可用性检查(lazy init + 显式 disable),而不是在测试里设私有属性。

关联下一个意见(_run_dump_to_db 间接层)可以一并清理。

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in a2bc255. Removed _backend_available, _ensure_backend_available, and the _run_dump_to_db indirection. Backend unavailability is now reported by SnapshotImportBackend.dump_to_db itself via ImportToolMissingError (caught at the import statement in snapshot_import_backend.py), so the private flag path is gone and test_import_raises_on_missing_tool now monkeypatches service._backend.dump_to_db directly.

Comment thread src/pt_snap_cli/core/import_service.py Outdated
self._focus_service = focus_service
self._backend = backend or SnapshotImportBackend()
self._backend_available = True
self._run_dump_to_db: Callable[[Path, Path, int | None], Path] | None = (

@codeasier codeasier Jun 21, 2026 •

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 意见 2: _run_dump_to_db 间接层无价值

ImportService.__init__ 把自己持有的 self._backend.dump_to_db 又包成 self._run_dump_to_db,所有调用点都用 self._run_dump_to_db(...)。这层包装没有抽象价值(只多了个可被 monkeypatch 的入口),只是为了配合前一条死代码分支而存在。

建议: 直接 self._backend.dump_to_db(...),把 backend 状态检查集中到 snapshot_import_backend.py 内部。

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed together with the previous comment. _run_dump_to_db indirection removed; the service now calls self._backend.dump_to_db(...) directly. The assert self._run_dump_to_db is not None is also gone.

Comment thread src/pt_snap_cli/core/import_service.py Outdated
focus_state = None
if options.set_focus:
if self._focus_service is None:
from pt_snap_cli.core.focus_service import FocusService

@codeasier codeasier Jun 21, 2026 •

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 意见 10: 这里的 lazy import 既不必要也无收益

focus_service 早已能正常 import,不会产生循环引用。建议直接放到模块顶部,简化控制流 — 读到这里的人会怀疑有循环依赖,但其实没有。

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed (same fix as the related comment on FocusService lazy import). FocusService is now imported at module top in import_service.py; there is no cycle.

if path.suffix.lower() not in VALID_SUFFIXES:
raise SnapshotFileInvalidError(
f"Snapshot suffix must be .pkl or .pickle, got {path.suffix!r}"
)

@codeasier codeasier Jun 21, 2026 •

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 意见 4: pickle.load 在 backend 之前完整反序列化一次,backend 又会再 load 一次

同一个 pickle 被读两遍:

  • 第一次:pickle.load 完整反序列化(只为验证 isinstance(obj, dict))
  • 第二次:vendor dump() 内的 load_pickle_to_dict 又 pickle.load 一次

对 11 MB fixture 没问题,但生产环境(GB 级 snapshot)会有不必要的内存峰值和延迟。vendor 的 load_pickle_to_dict(file_util.py:62)已经做了 dict 类型校验,本层重复校验属于重复工作。

建议: 移除 wrapper 的 pickle.load,只校验存在 + 后缀 + 可读;或保留为快速 fail-fast,但把 is_dict 校验放到 backend 内部短路以避免重复反序列化。

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed. The wrapper's pickle.load is gone — _validate_snapshot_file now only checks existence, file-ness, and suffix. The vendored load_pickle_to_dict still raises a clear error on bad pickles, and that error propagates as ImportExecutionError from SnapshotImportBackend.dump_to_db. The duplicate full unpickle of multi-GB snapshots is avoided.

output_dir: Path,
device: int | None = None,
) -> Path:
db_path = output_dir / f"{snapshot_file.name}.db"

@codeasier codeasier Jun 21, 2026 •

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 意见 9: db_path 在两处独立计算,容易脱节

db_path = output_dir / f"{snapshot_file.name}.db" 算出的路径,和 vendor run_dump_to_db(snapshot2db.py:run_dump_to_db → 内部 Path(resolved_dump_dir) / f"{Path(snapshot_file).name}.db")算出的路径碰巧一致。两处独立计算,任何一处变更就会脱节,且 backend 拿到的是预期路径,不是 vendor 实际写出的路径(后端若有改动命名,这里就报错成"文件未生成")。

建议: 让 vendor 上游返回真实 db_path,或者把 db_path 作为参数从 adapter 传入并由 vendor 信任。不能让两处各自推导路径。

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed. db_path is now computed exactly once in SnapshotImportBackend.dump_to_db from output_dir / f"{snapshot_file.name}.db" — the same expression the vendored closure uses internally — and is the single value passed to _ensure_fresh_db and the post-write existence check. Both layers agree on the path by construction.

db_path = output_dir / f"{snapshot_file.name}.db"

try:
ok = run_dump_to_db(

@codeasier codeasier Jun 21, 2026 •

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ 关联意见 6: except Exception 过宽

except Exception as exc: 会吞掉所有异常(包括 KeyboardInterrupt 之外的系统异常如 MemoryError)。在 import 这种重 IO + 大反序列化操作里,保留原始 traceback 比包成 ImportExecutionError 更有诊断价值。

建议: 至少留一条 logger.exception(...) 保留原始 traceback,或者拆成 (ImportError, OSError, pickle.UnpicklingError, sqlite3.DatabaseError) 等已知失败类型。

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed. except Exception is replaced with a narrow tuple: (OSError, sqlite3.DatabaseError, pickle.UnpicklingError, ValueError). The comment in the backend explains the rationale — anything else (MemoryError, KeyboardInterrupt, …) is left untouched so the original traceback is preserved.

def __init__(self, path: str):
super().__init__(path, auto_create=True, with_dictionary_table=True)
# 清理旧版本表格
self._clear_old_tables()

@codeasier codeasier Jun 21, 2026 •

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 意见 8: _clear_old_tables 在 __init__ 静默 DROP TABLE

auto_create=True + 没有读取已有数据的代码,这个 _clear_old_tables 对已有 .db 文件会无声丢失数据。import 流程在 PR 默认路径下没问题(output_dir / f"{snapshot_file.name}.db" 是新文件),但如果用户对已有 db 重跑 import(常见于反复重导同一 fixture),数据会直接被清掉。

建议:

  • 加 if overwrite: bool = False 参数,默认 False 时直接报错
  • 或者在 CLI 层加 warning: "target db exists, will overwrite"
  • 至少把 DROP TABLE 改成 DROP TABLE IF EXISTS 并打 log(目前已经是 IF EXISTS,但没有 log)

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed. The safeguard now lives in SnapshotImportBackend._ensure_fresh_db: the adapter pre-deletes an existing target .db with a logged warning before delegating to the vendor, so the silent DROP in the vendored SnapshotDb.__init__ never touches pre-existing user data. The vendored snapshot_db.py itself is unchanged (preserving the byte-for-byte sync strategy from ATTRIBUTION.md).

logger.setLevel(level)

# 避免重复添加handler
if logger.hasHandlers():

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ Logger 副作用警告: get_logger 会对同名已有 logger handlers.clear(),这是对 stdlib logging 的破坏性操作。如果调用方进程里同名 logger 被其他库使用(很常见),会被这个 vendor 函数静默清掉 handlers。\n\n属于上游历史包袱,标记备忘,本 PR 不必改。


assert result.focus_state is not None
assert result.focus_state.source == "project"

@codeasier codeasier Jun 21, 2026 •

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 意见 11: test_import_snapshot_file_invalid_suffix 与 test_import_snapshot_file_invalid_corrupt_pickle 测试体重复

suffix 测试用 foo.txt 后缀写 b"not a pickle",corrupt 测试用 corrupt.pkl 后缀写完全相同的 b"not a pickle"。两者覆盖的错误分支完全不同,但测试体一模一样(后者会先通过后缀检查,再走 pickle.load 失败分支;前者直接在后缀检查就拒了)。

建议: 把 corrupt 测试改成 corrupt.pkl 后缀 + 真坏的 pickle 数据(如 b"\x80\x04\x95not pickle" 或 binary garbage),让两个测试覆盖真实不同的失败路径。

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed. The two tests now cover distinct paths: test_import_snapshot_file_invalid_suffix writes a .txt file (the suffix check fires and the backend is never invoked); test_import_snapshot_file_invalid_corrupt_pickle writes pickle-protocol garbage with .pkl suffix and confirms the vendored load_pickle_to_dict failure surfaces as ImportExecutionError. The test name and body now match the path each one covers.

Comment thread pyproject.toml
extend-exclude = ["src/pt_snap_cli/vendor"]

[tool.ruff.lint]
select = ["E", "F", "W", "I", "N", "UP", "B", "C4"]

@codeasier codeasier Jun 21, 2026 •

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Blocker 3: vendor 子树缺少 LICENSE / NOTICE / 来源元数据

src/pt_snap_cli/vendor/memsnapdump/ 下 23 个 Python 文件原样复制自上游,但:

  • 没有 LICENSE-memsnapdump / NOTICE / ATTRIBUTION.md 标注来源
  • 没有上游 commit SHA / tag pin(本 PR 用什么版本?)
  • 没有上游 license 文本(Apache 2.0 / MIT?决定 pt-snap-cli 是否需要兼容声明)

强制要求: 在 src/pt_snap_cli/vendor/memsnapdump/ 下添加:

  • ATTRIBUTION.md: 写明上游仓库 URL + commit SHA + license + 同步策略
  • LICENSE: 上游 license 原文(项目 license 是 MIT,与上游协议需确认兼容)

CI 还可以加一个对账 step:

git log --oneline src/pt_snap_cli/vendor/memsnapdump/ | head -5
# 提醒:vendor 自上次同步后是否变化

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed. Added src/pt_snap_cli/vendor/memsnapdump/ATTRIBUTION.md (upstream URL, pinned SHA 87ea207372e0985790e1a28dab499dccf3c3b9a4, license status, sync strategy) and src/pt_snap_cli/vendor/memsnapdump/LICENSE (MIT, same as pt-snap-cli). The upstream MemSnapDump repo currently has no top-level LICENSE file — both projects share the same author and the closure is taken under the same MIT terms. When upstream publishes a formal LICENSE file, the SHA in ATTRIBUTION.md should be bumped and the upstream text should replace the local one.

Comment thread src/pt_snap_cli/cli.py Outdated
ImportToolMissingError,
ImportExecutionError,
SnapshotFileInvalidError,
FocusFileInvalidError,

@codeasier codeasier Jun 21, 2026 •

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 意见 6: DatabaseSchemaError 在 import 异常列表里语义不匹配

DatabaseSchemaError 来自 FocusService._validated_context(import 写 focus 时打开新 DB 校验 schema)。它在 import 流程里确实会抛,但放在 CLI 层混在 import 专属错误里,异常来源被混在调用方,调用方读起来不知道是谁抛的。

建议: 让 ImportService 把这类错误包进 ImportExecutionError,或在 docstring 显式声明"可能传播的外部异常"。

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed. DatabaseSchemaError (and FocusFileInvalidError) raised during focus write are now caught inside ImportService.import_snapshot and re-raised as ImportExecutionError with a clear message. The CLI only needs to catch the three import-layer exception types and never sees the focus/database domain errors leak through.

Comment thread src/pt_snap_cli/cli.py Outdated
_error(str(e))

typer.echo(f"Imported: {result.db_path}")
if result.focus_state is not None:

@codeasier codeasier Jun 21, 2026 •

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 意见 3: getattr(result.focus_state, "focus_file", None) 是无意义兜底

FocusState 字段 focus_file 是固定属性(已在 focus_service.py:78-83 验证)。用 getattr(..., None) 看似防御,实则掩盖未来字段重命名时的破坏。

建议: 直接 result.focus_state.focus_file if result.focus_state else None。

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed. getattr(result.focus_state, "focus_file", None) is gone; the CLI now does result.focus_state.focus_file if result.focus_state is not None and result.focus_state.focus_file is not None. focus_file is a stable field on FocusState (see focus_service.py:78-83).

pass


class ImportToolMissingError(PtSnapCoreError):

@codeasier codeasier Jun 21, 2026 •

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 设计确认: ImportToolMissingError 的 docstring 把语义从"用户漏装 memsnapdump" 改为"vendor 打包异常",解释清楚。

但前面关于 _backend_available 死代码的意见会导致 ImportToolMissingError 在生产路径上实际上触发不到(除非 vendor import 失败 — 但那个分支在 snapshot_import_backend.py 的 try/except 里独立处理,不会到这里)。两个错误层抢同一类语义,实际只有一层会被触发。

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Acknowledged. The ImportToolMissingError semantics now match its docstring: it is only raised from snapshot_import_backend.py's ImportError branch (i.e. the vendored closure could not be imported — a packaging problem). With the dead _backend_available path removed, there is exactly one place that raises this error in production, not two competing ones.

Comment thread tests/test_cli.py
"""Test import rejects non-pkl snapshot paths."""
snapshot = tmp_path / "sample.txt"
snapshot.write_text("not a snapshot")

@codeasier codeasier Jun 21, 2026 •

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 意见 12: test_import_invalid_suffix 用 write_text("not a snapshot") 但 .txt 后缀实际触发的是 pickle.load 失败,不是后缀检查

write_text 写 "not a snapshot" 是合法字符串(可被 pickle 部分解析),而 .txt 后缀先被后缀检查拒掉。

但若改用 write_bytes(b"corrupt garbage") + .pkl 后缀,反而不会进入后缀检查路径,而是触发 pickle.load 失败 — 也就是测试标题('invalid suffix')与实际触发的检查('can't unpickle')不一致。

建议: 测试名改成 test_import_invalid_suffix_or_corrupt,或拆成两个独立测试,或用 write_bytes(b"\x80garbage") 配 .pkl 后缀让 pickle.load 真失败。

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed together with the test split above. The CLI suffix test now uses .txt (suffix check rejects) and a separate corrupt-pickle path is exercised in tests/core/test_import_service.py.

Comment thread pyproject.toml
extend-exclude = '''
(
^/src/pt_snap_cli/vendor/
)

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 black extend-exclude 配置正确: vendor 子树被排除在 black 重写之外,避免未来 vendor 同步时被反向风格化。这一条是 vendor 模式的关键防线。

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Acknowledged. extend-exclude is intentionally kept on both black and ruff so future re-syncs of the vendor closure do not get reformatted on every CI pass. ATTRIBUTION.md documents this contract.

Comment thread pyproject.toml
[tool.black]
line-length = 100
target-version = ["py310"]
extend-exclude = '''

@codeasier codeasier Jun 21, 2026 •

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Blocker 4: [tool.setuptools.package-data] 不包含 vendor 子树,需要验证 wheel 完整性

pt_snap_cli = ["query/templates/*.yaml", "query/templates/*/*.yaml"] 只显式列了 YAML 模板(本 hunk 之前的几行)。vendor 目录下的 .py 文件没被显式列出。

虽然 [tool.setuptools.packages.find] 设了 include = ["pt_snap_cli*"],理论上会通过模块发现把 vendor 子包打进去 — 但这条路径没有任何 CI step 验证过。

最危险的失败模式:

  • pip install -e . 走 develop 模式,vendor 文件存在(本地 import 一切正常)
  • pip install dist/*.whl 装正式包,如果 setuptools.find 没把 vendor/... 收进去 → ImportToolMissingError 在生产环境触发
  • 本地 CI pytest 走 develop 模式,这个问题永远抓不到

强制要求: 在 CI 加 step:

pip install build
python -m build --wheel
pip install dist/*.whl --force-reinstall
pt-snap import tests/fixtures/snapshots/snapshot_with_empty_cache.pkl --no-focus

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed. The wheel smoke test is now wired into CI (only on Python 3.13 to keep the matrix cost down): it builds dist/*.whl, force-reinstalls it, asserts pt_snap_cli.vendor.memsnapdump.tools.adaptors.snapshot2db.run_dump_to_db is importable, and runs pt-snap import against the real snapshot_with_empty_cache.pkl fixture — verified locally to produce the same 11 MB .db artifact. Any future packaging regression that drops the vendor closure will fail CI immediately instead of only surfacing in production.

codeasier added 2 commits June 22, 2026 07:39
Backend cleanup:
- Remove _backend_available / _ensure_backend_available / _run_dump_to_db
  indirection in ImportService. Vendor unavailability is reported via
  ImportToolMissingError from SnapshotImportBackend itself, eliminating
  the dead private-flag path that was only exercised by tests.
- Drop the redundant pickle.load in _validate_snapshot_file. The vendored
  load_pickle_to_dict already validates dict type and surfaces a clearer
  error, avoiding a second full unpickle of multi-GB snapshots.
- Move the FocusService import to the module top — there is no import
  cycle, and the lazy import only confused readers.
- Tighten snapshot_import_backend: compute db_path once (matching the
  vendor closure's expression), narrow exception handling to specific
  failure modes (OSError, sqlite3, pickle, ValueError), pre-delete any
  existing target .db with a logged warning to prevent the vendored
  SnapshotDb from silently DROPping tables in pre-existing databases.
- Wrap DatabaseSchemaError / FocusFileInvalidError from focus writes as
  ImportExecutionError so callers only see one import-layer exception
  type from the service.

CLI:
- Drop getattr(result.focus_state, 'focus_file', None) fallback in the
  import command — focus_file is a stable FocusState field.

Tests:
- Split corrupt-pickle test from suffix test: suffix test now uses a .txt
  file (only triggers suffix check); corrupt test writes pickle-protocol
  garbage with .pkl suffix and confirms the vendored backend surfaces
  ImportExecutionError.
- Update the missing-tool / upstream-failure tests to monkeypatch
  service._backend.dump_to_db instead of private flags.

Vendoring:
- Add src/pt_snap_cli/vendor/memsnapdump/ATTRIBUTION.md documenting the
  upstream repository, pinned SHA, license status, and resync strategy.
- Add src/pt_snap_cli/vendor/memsnapdump/LICENSE (MIT, same as pt-snap-cli).
- The upstream MemSnapDump repo has no top-level LICENSE file; both
  projects are by the same author and the closure is taken under MIT.

CI:
- Add a wheel smoke test that builds dist/*.whl, force-reinstalls it,
  confirms the vendored closure is importable, and runs pt-snap import
  against a real fixture. This catches any setuptools packaging
  regression that would otherwise only surface in production installs.

Refs review comments on #67.
@codeasier
codeasier merged commit 7a40d5c into main Jun 22, 2026
5 checks passed
@codeasier
codeasier deleted the feat/memsnapdump-vendoring branch June 22, 2026 09:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE] 集成 MemSnapDump 的原始快照转 DB 能力

1 participant