Repository navigation
perf: deduplicate SnapshotDB callstacks and skip Frame rebuild in split - #114
Conversation
…it (#113) Import stored the full callstack text on every trace event, so a snapshot with a few thousand distinct callstacks inflated into gigabytes of duplicated text and made SQLite writing dominate import time. Interning the text and storing an integer reference removes that cost without touching allocator replay. `trace_entry_<device>` now stores `callstackId` referencing a shared `callstack` table. Replay, synthetic event and block identities, totals, and callstack text are unchanged: every trace and block row matches the pre-change output on all five deserializable fixtures once `callstackId` is resolved. `SplitService` also replayed with eager `Frame` construction even though slices serialize the original frame dictionaries, so it rebuilt objects only to discard them. Measured on the sanitized benchmark fixtures (628k events / 175 MB source): import 29.1s -> 8.6s, database 5.95 GB -> 177 MB, peak RSS 5.8 GB -> 814 MB. Splitting the 131k fixture into 4 slices goes 12.5s -> 2.5s with identical canonical slice hashes. `IMPORT_FORMAT_VERSION` moves to 2 so cached databases in the old layout are rebuilt rather than reused.
codeasier
left a comment
There was a problem hiding this comment.
交叉评审结论(3 个独立评审器 + 裁判会话核实)
- 评审器:
bailian/deepseek-v4-flash-0731(3 项发现)、unraid-wg/gemini-3.8-flash(0 项)、opencode-go/muse-spark-1.3-contributor(4 项,1 项被驳回)。全部成功;每条候选发现均由裁判会话对照代码与实测逐条核实,非投票决定。 - PR 核心收益成立:导入提速、34x 数据库压缩、调用栈查询加速、五组 fixture 逐行等价性均有据可查。
- 核实后共 4 项发现:1 中、2 低、1 提示,已逐条行内标注。
- 驳回 1 项(为
callstackId建索引):无任何打包查询受益——GROUP BY需要每行size(非覆盖索引无法避免全表扫描),三个模板的 JOIN 均通过主键(t.id、callstack.id)解析;加索引反而增加导入耗时与库体积,与 PR 目标相悖。 - 实证手段:在仓库自带脱敏 fixture(与 PR 基准相同的 131k/628k 输入)上运行只读探针测量 interner 缓存增长;用真实查询管线(
Context+QueryExecutor)在 v1 布局数据库上复现event模板失败。 - 遗留测试缺口:npu workspace 非空帧的 split 测试、旧布局数据库的调用栈模板测试、interner 缓存规模有界测试。
| callstack_id = existing | ||
| self._by_frames[frame_key] = (frames, callstack_id) | ||
|
|
||
| self._by_container[id(frames)] = (frames, callstack_id) |
There was a problem hiding this comment.
[中] _by_container 按 O(事件数) 增长,并把每个帧容器钉住到整个 dump 结束
实测(仓库自带脱敏 fixture,即 PR 基准输入,只读探针):
| 样本 | 事件数 | 不同容器 | 不同帧键元组 | 不同文本 |
|---|---|---|---|---|
| 628k | 628,837 | ~421,115 | 4,334 | 3,770 |
| 131k | 131,275 | ~87,550 | 883 | 722 |
pickle memo 共享的是单个帧字典(因此 _by_frames 命中率约 99.3%),而非外层容器——docstring 第 9 行「the pickle memo already shares the frame containers」与实测不符。_by_container 只能命中约 33% 的查询,却在 628k dump 上额外保留约 100 MB(dict 条目 + tuple + 被钉住的 list)。导入路径以 _raw_frames=True 回放(snapshot2db.py:199),且回放逐个弹出事件(simulate.py:176),本可渐进释放的帧列表被 interner 钉住——这与 PR 中 Linux 复测分支比基线高约 67 MB 相一致。
建议:去掉 _by_container,仅保留 _by_frames——其「每 key 钉住一个样本容器」的机制已保证 id 复用安全性(样本使 key 中的帧字典保持存活,test_reclaimed_container_address_does_not_produce_a_false_hit 覆盖的性质不受影响);或对该缓存设置上限。并补充一个「N 个不同容器共享同一文本时缓存规模有界」的测试。
| _, replayed = replay_snapshot(representation, device, hooker=hooker) | ||
| # Slices serialize the original frame dictionaries, so eager Frame | ||
| # objects would be rebuilt only to be discarded. | ||
| _, replayed = replay_snapshot(representation, device, hooker=hooker, _raw_frames=True) |
There was a problem hiding this comment.
[低] _raw_frames=True 使 torch-npu workspace 块在切片中丢失帧
simulate.py:124-131 的 _adapt_workspace_snapshot 构造 Block 时只传 frames=existed_seg.frames——raw 模式下它是 [](entities.py:241-242)——且未传播 _raw_frames,于是 Block.to_dict() 序列化为 "frames": [](entities.py:196-200)。PR 之前 eager 路径经 Frame._origin 能回传真实帧。该适配块经 slice_dump/hooker.py:99-101 深拷贝进入切片文件。
不影响回放有效性与 DB 导入(block2record 无帧列),也不崩溃,但 npu workspace 快照(首事件为 workspace_snapshot)的切片相对变更前静默丢数据。其余所有构造点均正确传播 _raw_frames(Block.build_from_event、Segment.build_from_event、snapshot_mutator.py:267-296),此处是唯一遗漏。
修复:Block 构造补传 _raw_frames=existed_seg._raw_frames;现有 test_workspace_marker_splits_and_replays 用的正是 "frames": [],无法捕获该问题,需补一个带非空 workspace segment 帧的 split 测试。
| active, | ||
| reserved, | ||
| ( | ||
| SELECT callstack FROM callstack |
There was a problem hiding this comment.
[低] 旧布局数据库上此模板硬失败:no such table: callstack
用真实管线(Context + QueryExecutor)在 v1 布局(内联 callstack TEXT)数据库上实测复现:QueryExecutionError: Query execution failed: no such table: callstack。此处此前是 SELECT *,与布局无关;callstack_analysis 与 active_memory_callstack_at_event 在 v1 库上同样失败(no such column: callstackId)。
问题链路:Context 仍只校验 dictionary 表(context.py:69-80),数据库能正常打开,错误推迟到查询时才以裸 SQLite 错误暴露、无重导入提示;而 docs/en/database.md:51-52(及 zh 镜像)仍声称遗留/外部数据库「remain queryable」。第一方缓存由 IMPORT_FORMAT_VERSION 1→2 触发重建保护,残留影响面是仅持有旧 .db(无源 pickle)的用户与外部/遗留数据库。目前没有任何测试对旧布局 fixture 执行涉及调用栈的模板(旧布局套件只用 leak_detection/required_query/size_query 一类)。
建议:渲染前探测缺列/缺表时给出友好的 schema 不匹配提示(引导重新导入)或做兼容回退;同步修订 docs 中该句;补一个锁定该行为的测试。
| HAVING alloc_count >= {{ min_count }} | ||
| AND total_size >= {{ min_size }} | ||
| ) AS grouped | ||
| JOIN callstack c ON c.id = grouped.callstackId |
There was a problem hiding this comment.
[提示] INNER JOIN 会静默丢弃悬空的 callstackId 分组
此处 JOIN callstack c 为内连接:若某分组聚合后无法在 callstack 表解析到行,该分组直接消失,而非以 NULL / 占位文本呈现。兄弟模板 active_memory_callstack_at_event 对同类语义用的是 LEFT JOIN … COALESCE(cs.callstack, '[missing callstack]')。
第一方导入的覆盖性有保证且已有测试(test_callstack_table_is_deduplicated_and_covers_every_event);WHERE callstackId IS NOT NULL 对 NULL 的处理也与旧版 WHERE callstack IS NOT NULL 一致。该场景仅在「外部按新 schema 生成且存在悬空引用」时可达,属低概率边界,仅为与兄弟模板的对称性考虑——可改为 LEFT JOIN。
codeasier
left a comment
There was a problem hiding this comment.
跟进轮交叉评审结论(3 个独立评审器 + 裁判会话实证核实)
针对修复提交 6143228(fix: address PR 114 review feedback)的跟进验证。
- 上一轮 4 项发现全部解决,未发现回归:
- F1 [中] interner O(事件数) 钉住 →
_by_container已完整移除;仅剩_by_frames(键为帧 id 元组,每键钉住一个样本容器,键中全部帧 dict 保持存活 → id 复用安全仍成立;空帧键()天然安全)。缓存规模以不同调用栈为界(628k 样本:4,334 条)。 - F2 [低] npu workspace 块丢帧 →
simulate.py:130补传_raw_frames,raw/eager 两模式均正确,新测试锁定。 - F3 [低] 旧布局库裸 SQLite 错误 → 实测
event/callstack_analysis/active_memory_callstack_at_event三个模板在 v1 库上全部命中友好提示;docs 中英文已同步;新测试锁定。 - F4 [提示] INNER JOIN 丢分组 → 已改
LEFT JOIN。
- F1 [中] interner O(事件数) 钉住 →
- 裁判实证:干净状态下全量套件 766 passed, 1 xfailed(作者报告 764 + 净增 2 测试);治理与 fixture 溯源测试 28 全过。(首次全量出现的
test_api.py::test_set_focus_device_requires_database失败为预先存在的测试隔离泄漏——tests/core/test_import_service.py:75未隔离 CWD 把 focus 写进主仓祖先.pt-snap/focus.json——与本 PR 改动无关,建议另开 issue。) - 残留项(均不阻塞合并,已行内标注):1 低(PROVENANCE.md 未为
6143228追加条目且现存条目描述已失真)、1 低(executor 子串匹配对限定列名脆弱 + 分支测试缺口)、3 提示(docstring 失真、interner 安全机制失去直接测试、悬空分组显示为 "None")。 - 评审器:
bailian/deepseek-v4-flash-0731、unraid-wg/gemini-3.8-flash、opencode-go/muse-spark-1.3-contributor,全部成功,结论一致。
总评:修复质量高,4/4 完整解决且实证无回归;残留项均为文档/测试加固类,可合并后跟进。
| inlined `callstack` text column, and a shared device-suffix-free `callstack` | ||
| table holds the text. `TraceEntry.callstack_frames()` exposes the frame | ||
| container that already backed `get_callstack()` so interning can key on | ||
| container identity; the interner holds a reference to every keyed container so |
There was a problem hiding this comment.
[低] 本条目描述已与最终实现不符,且 6143228 未追加新条目(3 个评审器一致指出)
本条目(a7e9f97 写入)仍称 interning 以「container identity」为键、「holds a reference to every keyed container」——修复提交已移除容器身份快路径与逐事件钉住,最终实现以帧身份元组 tuple(map(id, frames)) 为键、每个不同键仅钉住一个样本容器。且 6143228 修改了 snapshot 运行时文件(simulate.py、callstack.py)却未按本文件自身的追加策略(每次更新须追加条目)补记。
已核实 CI 守卫按 base/head blob 比较仍通过(PR 整体确有 PROVENANCE 变更),不构成守卫违规,属内容准确性缺口。建议合并前追加一条日期条目,记录:帧身份键 + 单样本钉住、workspace 块 _raw_frames 传播、executor 友好错误映射。
| @staticmethod | ||
| def _execution_error(error: sqlite3.OperationalError) -> QueryExecutionError: | ||
| detail = str(error) | ||
| if "no such table: callstack" in detail or "no such column: callstackId" in detail: |
There was a problem hiding this comment.
[低] 子串匹配对限定列名(qualified column)脆弱
"no such column: callstackId" 无法匹配 no such column: t.callstackId 或 no such column: trace_entry_0.callstackId(t. 直接跟在冒号空格后,子串不成立)。
当前惰性(裁判实测):真实 v1 库上三个调用栈模板都先触发缺表错误 no such table: callstack 而命中友好提示——包括 active_memory_callstack_at_event(其 ON t.callstackId = cs.id 的缺列错误永远后于 LEFT JOIN callstack 的缺表错误)。但一次模板改写(把裸列名改成限定引用)或一个「有 callstack 表但缺 callstackId 列」的混合布局库,即可让它静默退化为裸错误。
廉价加固:改用正则 re.search(r"no such column: (?:[\w.]+\.)?callstackId", detail),或直接匹配 "callstackId" in detail。
配套测试缺口:test_legacy_callstack_schema_error_is_actionable 只用合成 SQL SELECT callstack FROM callstack 锁定了缺表分支;callstack_analysis 在 v1 库上实际命中的缺列分支(no such column: callstackId)与 count() 路径均未测。
| """Assign one integer id per distinct callstack text. | ||
|
|
||
| PyTorch snapshots repeat a small number of callstacks across a very large | ||
| number of events, and the pickle memo already shares the frame containers |
There was a problem hiding this comment.
[提示] 此句与实测不符,且与下方已改写的段落自相矛盾
「the pickle memo already shares the frame containers behind them」——上一轮实测(628k 脱敏 fixture):628,837 个事件产生约 421,115 个不同外层容器,pickle memo 共享的是单个帧字典(因此 _by_frames 命中率约 99.3%),而非外层容器(这正是 _by_container 仅命中约 33% 的原因)。本段下方的查找段落已改写为按帧身份查找,此句宜一并修正(例如改为「the pickle memo already shares the individual frame objects behind them」)。
| assert interner.records() == [{"id": 0, "callstack": ""}] | ||
|
|
||
|
|
||
| def test_distinct_frame_identities_do_not_produce_a_false_hit(): |
There was a problem hiding this comment.
[提示] 替代测试未覆盖样本钉住(exemplar pinning)这一安全机制本身
旧的 test_reclaimed_container_address_does_not_produce_a_false_hit 被删除后,id 复用隐患(本类唯一的安全不变量:「_by_frames 值中的样本容器必须钉住键中全部帧 dict」)失去了直接测试——新测试只验证两个存活对象不混叠([INNER]/[OUTER]),即使删掉 intern() 里的 (frames, callstack_id) 样本存储它也会通过。实现本身构造上正确(裁判核实),但建议用 weakref 钉住该不变量:例如 intern 一个事件后 weakref.ref(frames),intern 若干其他事件后断言 ref() 仍非 None(样本未释放)。
| HAVING alloc_count >= {{ min_count }} | ||
| AND total_size >= {{ min_size }} | ||
| ) AS grouped | ||
| LEFT JOIN callstack c ON c.id = grouped.callstackId |
There was a problem hiding this comment.
[提示] 悬空分组经结果映射后显示为字符串 "None"
LEFT JOIN 修复后悬空 callstackId 分组以 c.callstack = NULL 呈现,但经 ResultMapper(mapper.py:69,str 转换器 str(None))后会显示为字符串 "None"。兄弟模板 active_memory_callstack_at_event 用 COALESCE(cs.callstack, '[missing callstack]')。第一方导入库不可达此场景(覆盖性有测试保证),仅为一贸性对齐建议:COALESCE(c.callstack, '[missing callstack]') AS callstack。
📝 Description
Import stored the full callstack text on every trace event. Because a snapshot typically holds only a few thousand distinct callstacks across hundreds of thousands of events, that inflated into gigabytes of duplicated text and made SQLite writing dominate import time.
This PR interns the text and stores an integer reference.
trace_entry_<device>now hascallstackIdpointing at a new sharedcallstack(id, callstack)table.Allocator replay is deliberately left alone, so this is a storage-layout change rather than a semantics change.
Three parts:
CallstackInternerassigns one id per distinct callstack text. It tries the frame-container identity first, then the identities of the individual frames, and only renders text on a miss. Distinct containers that render to the same text collapse onto one id, socallstackIdand text stay one-to-one and grouping by id equals grouping by text.SplitServiceno longer rebuildsFrameobjects — slices serialize the original frame dictionaries, so eager construction built objects only to discard them. Now passes_raw_frames=True.get_insert_values_by_recordsresolved the per-column value map once per cell; now once per batch. Default insert batch moved 1000 → 10000, which is safe because import stages rows in a temporary database and only publishes after a successful replay.Results
Measured on macOS, Python 3.13.12, against the sanitized benchmark fixtures.
Splitting the 131k fixture into 4 slices: 12.47s → 2.49s (5.0x), with identical
canonical_snapshot_sha256for every slice.Callstack storage on the 628k sample drops from 5,360 MB inline to 41.9 MB across 3,771 rows — a 128x dedup factor.
调用栈查询与聚合性能复测(628k 样本,Linux / Python 3.13.11)
调用栈去重不仅压缩了数据库体积,查询与聚合分析也全面变快,未发现任何回退场景。SQLite 机制下,旧版每行内联约 8.5 KB 的调用栈文本,几乎每条 trace 记录都要跨越 2~3 个 4096 字节溢出页(Overflow Pages),全表扫描需读取 5.67 GB;新版
trace_entry_<device>仅含整型列(全表约 50 MB,可整体驻留缓存),聚合模板先按整型callstackId分组过滤,最后仅对聚合后的少量去重行按主键 JOIN 文本表:callstack_analysis(GROUP BY callstack)active_memory_callstack_at_eventLIKE '%...%'(IN 子查询)WHERE callstack = ...vscallstackId = ...event(id=204412)event(LIMIT 1000)allocator_gapCompatibility
IMPORT_FORMAT_VERSIONmoves 1 → 2, so databases cached in the old layout reportimport_format_changedand are rebuilt rather than reused. Existing external databases with an inlinecallstackcolumn remain queryable viaContext(which only validatesdictionary), but the packaged callstack templates now expect the new layout.Not in this PR
Issue #113 also proposes replacing the structural replay in the import path with prefix sums plus hash-map address matching (a further ~2.5x). That is a correctness-sensitive change to replay itself and is intentionally left as a follow-up so it can be reviewed on its own. The structural replay stays as-is here, and it remains required for
split.🔗 Related Issue
Partially addresses #113
🧪 Testing
pytest: 764 passed, 1 xfailed (baseline onmainwas 756 passed, 1 xfailed; this PR adds 8 tests).ruff check,black --check, andbasedpyrightall clean on tracked files.Row-by-row equivalence against the pre-change implementation. I built databases from
mainand from this branch in separate subprocesses and compared the trace and block tables, reconstructing the inline callstack view through the join:snapshot_with_multi_devices.pklsnapshot_expandable.pklsnapshot_with_empty_cache_expandable.pklsnapshot_import_131k_sanitized.picklesnapshot_import_628k_sanitized.pickleAlso verified:
nonempty_callstacksgolden counts intests/snapshot/golden_observations.pyare unchanged, now read through the join.SCAN trace_entry_0 USING COVERING INDEX idx_trace_entry_0_allocated). I checked a view-based variant first and rejected it because it degraded that plan to a full scan plus temp B-tree.tests/snapshot/test_callstack_interner.pycovers text equality withget_callstack(), shared containers, distinct containers with equal frames, empty frames, eager vs raw frame paths, and that a keyed container cannot have its address reused to produce a false cache hit.test_callstack_table_is_deduplicated_and_covers_every_eventasserts one row per distinct text and that every event resolves.Skills fixture YAML still authors callstacks as text per trace row; the harness interns them so no fixture files changed.
✅ Checklist
docs/en/database.mdanddocs/zh/database.mdboth document thecallstackIdcolumn, the newcallstacktable with its uniqueness/scope/coverage guarantees, and how to join for the text.Snapshot Runtime Provenance
Snapshot provenance decision: updated
Snapshot provenance no-update reason:
📋 Additional Notes
callstackintentionally carries no device suffix because callstacks are shared across devices, and one interner spans the whole dump so multi-device databases share rows. The name also has to stay outside thetrace_entry_%space thatContext._discover_device_idsscans — an earlier draft usedtrace_entry_raw_0for a helper table and it was silently picked up as "device 0".The interner keys on
id(), which is only meaningful while the keyed object is alive, so it holds a reference to every keyed container. There is a regression test for the address-reuse hazard.