Skip to content

perf: deduplicate SnapshotDB callstacks and skip Frame rebuild in split - #114

Merged
codeasier merged 3 commits into
mainfrom
perf/import-callstack-dedup
Sep 6, 2026
Merged

codeasier merged 3 commits into
mainfrom
perf/import-callstack-dedup

Conversation

@codeasier

@codeasier codeasier commented Sep 5, 2026 •

Copy link
Copy Markdown
Owner

📝 Description

Import stored the full callstack text on every trace event. Because a snapshot typically holds only a few thousand distinct callstacks across hundreds of thousands of events, that inflated into gigabytes of duplicated text and made SQLite writing dominate import time.

This PR interns the text and stores an integer reference. trace_entry_<device> now has callstackId pointing at a new shared callstack(id, callstack) table.

Allocator replay is deliberately left alone, so this is a storage-layout change rather than a semantics change.

Three parts:

  1. Callstack interning — new CallstackInterner assigns one id per distinct callstack text. It tries the frame-container identity first, then the identities of the individual frames, and only renders text on a miss. Distinct containers that render to the same text collapse onto one id, so callstackId and text stay one-to-one and grouping by id equals grouping by text.
  2. SplitService no longer rebuilds Frame objects — slices serialize the original frame dictionaries, so eager construction built objects only to discard them. Now passes _raw_frames=True.
  3. Two write-path fixes found while profiling — get_insert_values_by_records resolved the per-column value map once per cell; now once per batch. Default insert batch moved 1000 → 10000, which is safe because import stages rows in a temporary database and only publishes after a successful replay.

Results

Measured on macOS, Python 3.13.12, against the sanitized benchmark fixtures.

Sample Import time Database Peak RSS
131k events / 43 MB 3.05s → 1.59s (1.9x) 664 MB → 31 MB (22x) 1.45 GB → 223 MB
628k events / 175 MB 29.1s → 8.59s (3.4x) 5.95 GB → 177 MB (34x) 5.81 GB → 814 MB

Splitting the 131k fixture into 4 slices: 12.47s → 2.49s (5.0x), with identical canonical_snapshot_sha256 for every slice.

Callstack storage on the 628k sample drops from 5,360 MB inline to 41.9 MB across 3,771 rows — a 128x dedup factor.

调用栈查询与聚合性能复测(628k 样本,Linux / Python 3.13.11)

调用栈去重不仅压缩了数据库体积,查询与聚合分析也全面变快,未发现任何回退场景。SQLite 机制下,旧版每行内联约 8.5 KB 的调用栈文本,几乎每条 trace 记录都要跨越 2~3 个 4096 字节溢出页(Overflow Pages),全表扫描需读取 5.67 GB;新版 trace_entry_<device> 仅含整型列(全表约 50 MB,可整体驻留缓存),聚合模板先按整型 callstackId 分组过滤,最后仅对聚合后的少量去重行按主键 JOIN 文本表:

查询场景 模板 / SQL 基线(main,内联文本) 本 PR(ID 去重) 提升
调用栈聚合分析 callstack_analysis(GROUP BY callstack) 26,065 ms 141.5 ms 184x
峰值时刻活跃块调用栈聚合 active_memory_callstack_at_event 190.5 ms 65.6 ms 2.9x
调用栈文本模糊检索 LIKE '%...%'(IN 子查询) 1,283.9 ms 60.3 ms 21.3x
按精确调用栈检索事件 WHERE callstack = ... vs callstackId = ... 924.4 ms 21.8 ms 42.3x
单事件点查(含还原文本) event(id=204412) 0.008 ms 0.007 ms 持平
读取 1,000 条事件(含文本) event(LIMIT 1000) 5.18 ms 3.19 ms 1.6x
批量导出 50,000 条(含文本) LIMIT 50000 323.4 ms 272.9 ms 1.2x
非调用栈数值表扫描 allocator_gap 329.1 ms 34.9 ms 9.4x

峰值 RSS 基线勘误:上表中 5.81 GB / 1.45 GB 的 RSS 基线为 macOS(Darwin / Python 3.13.12)实测。在 Linux 上复测同一基线 commit 89f5ddd,628k 事件峰值 RSS 已因 PR #85(_raw_frames=True)降至约 709 MB(131k 约 234 MB),与 PR #85 自报的优化后数据一致;本分支实测 776 MB,与基线基本持平(无恶化,也不存在 5.8 GB → 814 MB 的额外下降空间)。本 PR 的核心收益是 34x 数据库压缩、导入提速与上表的查询加速,而非峰值 RSS。

Compatibility

IMPORT_FORMAT_VERSION moves 1 → 2, so databases cached in the old layout report import_format_changed and are rebuilt rather than reused. Existing external databases with an inline callstack column remain queryable via Context (which only validates dictionary), but the packaged callstack templates now expect the new layout.

Not in this PR

Issue #113 also proposes replacing the structural replay in the import path with prefix sums plus hash-map address matching (a further ~2.5x). That is a correctness-sensitive change to replay itself and is intentionally left as a follow-up so it can be reviewed on its own. The structural replay stays as-is here, and it remains required for split.

🔗 Related Issue

Partially addresses #113

🧪 Testing

  • Tests added/updated
  • Manual testing completed

pytest: 764 passed, 1 xfailed (baseline on main was 756 passed, 1 xfailed; this PR adds 8 tests).

ruff check, black --check, and basedpyright all clean on tracked files.

Row-by-row equivalence against the pre-change implementation. I built databases from main and from this branch in separate subprocesses and compared the trace and block tables, reconstructing the inline callstack view through the join:

Fixture Trace rows differing Block rows differing
snapshot_with_multi_devices.pkl 0 / 8,134 0 / 3,219
snapshot_expandable.pkl 0 / 8,094 0 / 3,219
snapshot_with_empty_cache_expandable.pkl 0 / 8,134 0 / 3,219
snapshot_import_131k_sanitized.pickle 0 / 131,283 0 / 45,284
snapshot_import_628k_sanitized.pickle 0 / 628,839 0 / 217,535

Also verified:

  • The nonempty_callstacks golden counts in tests/snapshot/golden_observations.py are unchanged, now read through the join.
  • Numeric peak queries still use the covering index (SCAN trace_entry_0 USING COVERING INDEX idx_trace_entry_0_allocated). I checked a view-based variant first and rejected it because it degraded that plan to a full scan plus temp B-tree.
  • New tests/snapshot/test_callstack_interner.py covers text equality with get_callstack(), shared containers, distinct containers with equal frames, empty frames, eager vs raw frame paths, and that a keyed container cannot have its address reused to produce a false cache hit.
  • New test_callstack_table_is_deduplicated_and_covers_every_event asserts one row per distinct text and that every event resolves.

Skills fixture YAML still authors callstacks as text per trace row; the harness interns them so no fixture files changed.

✅ Checklist

  • Code follows project guidelines
  • Documentation updated (if applicable)
  • No new warnings introduced
  • Changes tested locally

docs/en/database.md and docs/zh/database.md both document the callstackId column, the new callstack table with its uniqueness/scope/coverage guarantees, and how to join for the text.

Snapshot Runtime Provenance

  • Snapshot provenance evaluation completed when snapshot runtime files change
    Snapshot provenance decision: updated
    Snapshot provenance no-update reason:

📋 Additional Notes

callstack intentionally carries no device suffix because callstacks are shared across devices, and one interner spans the whole dump so multi-device databases share rows. The name also has to stay outside the trace_entry_% space that Context._discover_device_ids scans — an earlier draft used trace_entry_raw_0 for a helper table and it was silently picked up as "device 0".

The interner keys on id(), which is only meaningful while the keyed object is alive, so it holds a reference to every keyed container. There is a regression test for the address-reuse hazard.

…it (#113)

Import stored the full callstack text on every trace event, so a snapshot with
a few thousand distinct callstacks inflated into gigabytes of duplicated text
and made SQLite writing dominate import time. Interning the text and storing an
integer reference removes that cost without touching allocator replay.

`trace_entry_<device>` now stores `callstackId` referencing a shared
`callstack` table. Replay, synthetic event and block identities, totals, and
callstack text are unchanged: every trace and block row matches the pre-change
output on all five deserializable fixtures once `callstackId` is resolved.

`SplitService` also replayed with eager `Frame` construction even though slices
serialize the original frame dictionaries, so it rebuilt objects only to discard
them.

Measured on the sanitized benchmark fixtures (628k events / 175 MB source):
import 29.1s -> 8.6s, database 5.95 GB -> 177 MB, peak RSS 5.8 GB -> 814 MB.
Splitting the 131k fixture into 4 slices goes 12.5s -> 2.5s with identical
canonical slice hashes.

`IMPORT_FORMAT_VERSION` moves to 2 so cached databases in the old layout are
rebuilt rather than reused.

@codeasier codeasier left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

交叉评审结论(3 个独立评审器 + 裁判会话核实)

  • 评审器:bailian/deepseek-v4-flash-0731(3 项发现)、unraid-wg/gemini-3.8-flash(0 项)、opencode-go/muse-spark-1.3-contributor(4 项,1 项被驳回)。全部成功;每条候选发现均由裁判会话对照代码与实测逐条核实,非投票决定。
  • PR 核心收益成立:导入提速、34x 数据库压缩、调用栈查询加速、五组 fixture 逐行等价性均有据可查。
  • 核实后共 4 项发现:1 中、2 低、1 提示,已逐条行内标注。
  • 驳回 1 项(为 callstackId 建索引):无任何打包查询受益——GROUP BY 需要每行 size(非覆盖索引无法避免全表扫描),三个模板的 JOIN 均通过主键(t.id、callstack.id)解析;加索引反而增加导入耗时与库体积,与 PR 目标相悖。
  • 实证手段:在仓库自带脱敏 fixture(与 PR 基准相同的 131k/628k 输入)上运行只读探针测量 interner 缓存增长;用真实查询管线(Context + QueryExecutor)在 v1 布局数据库上复现 event 模板失败。
  • 遗留测试缺口:npu workspace 非空帧的 split 测试、旧布局数据库的调用栈模板测试、interner 缓存规模有界测试。

callstack_id = existing
self._by_frames[frame_key] = (frames, callstack_id)

self._by_container[id(frames)] = (frames, callstack_id)

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[中] _by_container 按 O(事件数) 增长,并把每个帧容器钉住到整个 dump 结束

实测(仓库自带脱敏 fixture,即 PR 基准输入,只读探针):

样本 事件数 不同容器 不同帧键元组 不同文本
628k 628,837 ~421,115 4,334 3,770
131k 131,275 ~87,550 883 722

pickle memo 共享的是单个帧字典(因此 _by_frames 命中率约 99.3%),而非外层容器——docstring 第 9 行「the pickle memo already shares the frame containers」与实测不符。_by_container 只能命中约 33% 的查询,却在 628k dump 上额外保留约 100 MB(dict 条目 + tuple + 被钉住的 list)。导入路径以 _raw_frames=True 回放(snapshot2db.py:199),且回放逐个弹出事件(simulate.py:176),本可渐进释放的帧列表被 interner 钉住——这与 PR 中 Linux 复测分支比基线高约 67 MB 相一致。

建议:去掉 _by_container,仅保留 _by_frames——其「每 key 钉住一个样本容器」的机制已保证 id 复用安全性(样本使 key 中的帧字典保持存活,test_reclaimed_container_address_does_not_produce_a_false_hit 覆盖的性质不受影响);或对该缓存设置上限。并补充一个「N 个不同容器共享同一文本时缓存规模有界」的测试。

_, replayed = replay_snapshot(representation, device, hooker=hooker)
# Slices serialize the original frame dictionaries, so eager Frame
# objects would be rebuilt only to be discarded.
_, replayed = replay_snapshot(representation, device, hooker=hooker, _raw_frames=True)

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[低] _raw_frames=True 使 torch-npu workspace 块在切片中丢失帧

simulate.py:124-131 的 _adapt_workspace_snapshot 构造 Block 时只传 frames=existed_seg.frames——raw 模式下它是 [](entities.py:241-242)——且未传播 _raw_frames,于是 Block.to_dict() 序列化为 "frames": [](entities.py:196-200)。PR 之前 eager 路径经 Frame._origin 能回传真实帧。该适配块经 slice_dump/hooker.py:99-101 深拷贝进入切片文件。

不影响回放有效性与 DB 导入(block2record 无帧列),也不崩溃,但 npu workspace 快照(首事件为 workspace_snapshot)的切片相对变更前静默丢数据。其余所有构造点均正确传播 _raw_frames(Block.build_from_event、Segment.build_from_event、snapshot_mutator.py:267-296),此处是唯一遗漏。

修复:Block 构造补传 _raw_frames=existed_seg._raw_frames;现有 test_workspace_marker_splits_and_replays 用的正是 "frames": [],无法捕获该问题,需补一个带非空 workspace segment 帧的 split 测试。

active,
reserved,
(
SELECT callstack FROM callstack

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[低] 旧布局数据库上此模板硬失败:no such table: callstack

用真实管线(Context + QueryExecutor)在 v1 布局(内联 callstack TEXT)数据库上实测复现:QueryExecutionError: Query execution failed: no such table: callstack。此处此前是 SELECT *,与布局无关;callstack_analysis 与 active_memory_callstack_at_event 在 v1 库上同样失败(no such column: callstackId)。

问题链路:Context 仍只校验 dictionary 表(context.py:69-80),数据库能正常打开,错误推迟到查询时才以裸 SQLite 错误暴露、无重导入提示;而 docs/en/database.md:51-52(及 zh 镜像)仍声称遗留/外部数据库「remain queryable」。第一方缓存由 IMPORT_FORMAT_VERSION 1→2 触发重建保护,残留影响面是仅持有旧 .db(无源 pickle)的用户与外部/遗留数据库。目前没有任何测试对旧布局 fixture 执行涉及调用栈的模板(旧布局套件只用 leak_detection/required_query/size_query 一类)。

建议:渲染前探测缺列/缺表时给出友好的 schema 不匹配提示(引导重新导入)或做兼容回退;同步修订 docs 中该句;补一个锁定该行为的测试。

HAVING alloc_count >= {{ min_count }}
AND total_size >= {{ min_size }}
) AS grouped
JOIN callstack c ON c.id = grouped.callstackId

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[提示] INNER JOIN 会静默丢弃悬空的 callstackId 分组

此处 JOIN callstack c 为内连接:若某分组聚合后无法在 callstack 表解析到行,该分组直接消失,而非以 NULL / 占位文本呈现。兄弟模板 active_memory_callstack_at_event 对同类语义用的是 LEFT JOIN … COALESCE(cs.callstack, '[missing callstack]')。

第一方导入的覆盖性有保证且已有测试(test_callstack_table_is_deduplicated_and_covers_every_event);WHERE callstackId IS NOT NULL 对 NULL 的处理也与旧版 WHERE callstack IS NOT NULL 一致。该场景仅在「外部按新 schema 生成且存在悬空引用」时可达,属低概率边界,仅为与兄弟模板的对称性考虑——可改为 LEFT JOIN。

@codeasier codeasier left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

跟进轮交叉评审结论(3 个独立评审器 + 裁判会话实证核实)

针对修复提交 6143228(fix: address PR 114 review feedback)的跟进验证。

  • 上一轮 4 项发现全部解决,未发现回归:
    • F1 [中] interner O(事件数) 钉住 → _by_container 已完整移除;仅剩 _by_frames(键为帧 id 元组,每键钉住一个样本容器,键中全部帧 dict 保持存活 → id 复用安全仍成立;空帧键 () 天然安全)。缓存规模以不同调用栈为界(628k 样本:4,334 条)。
    • F2 [低] npu workspace 块丢帧 → simulate.py:130 补传 _raw_frames,raw/eager 两模式均正确,新测试锁定。
    • F3 [低] 旧布局库裸 SQLite 错误 → 实测 event / callstack_analysis / active_memory_callstack_at_event 三个模板在 v1 库上全部命中友好提示;docs 中英文已同步;新测试锁定。
    • F4 [提示] INNER JOIN 丢分组 → 已改 LEFT JOIN。
  • 裁判实证:干净状态下全量套件 766 passed, 1 xfailed(作者报告 764 + 净增 2 测试);治理与 fixture 溯源测试 28 全过。(首次全量出现的 test_api.py::test_set_focus_device_requires_database 失败为预先存在的测试隔离泄漏——tests/core/test_import_service.py:75 未隔离 CWD 把 focus 写进主仓祖先 .pt-snap/focus.json——与本 PR 改动无关,建议另开 issue。)
  • 残留项(均不阻塞合并,已行内标注):1 低(PROVENANCE.md 未为 6143228 追加条目且现存条目描述已失真)、1 低(executor 子串匹配对限定列名脆弱 + 分支测试缺口)、3 提示(docstring 失真、interner 安全机制失去直接测试、悬空分组显示为 "None")。
  • 评审器:bailian/deepseek-v4-flash-0731、unraid-wg/gemini-3.8-flash、opencode-go/muse-spark-1.3-contributor,全部成功,结论一致。

总评:修复质量高,4/4 完整解决且实证无回归;残留项均为文档/测试加固类,可合并后跟进。

inlined `callstack` text column, and a shared device-suffix-free `callstack`
table holds the text. `TraceEntry.callstack_frames()` exposes the frame
container that already backed `get_callstack()` so interning can key on
container identity; the interner holds a reference to every keyed container so

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[低] 本条目描述已与最终实现不符,且 6143228 未追加新条目(3 个评审器一致指出)

本条目(a7e9f97 写入)仍称 interning 以「container identity」为键、「holds a reference to every keyed container」——修复提交已移除容器身份快路径与逐事件钉住,最终实现以帧身份元组 tuple(map(id, frames)) 为键、每个不同键仅钉住一个样本容器。且 6143228 修改了 snapshot 运行时文件(simulate.py、callstack.py)却未按本文件自身的追加策略(每次更新须追加条目)补记。

已核实 CI 守卫按 base/head blob 比较仍通过(PR 整体确有 PROVENANCE 变更),不构成守卫违规,属内容准确性缺口。建议合并前追加一条日期条目,记录:帧身份键 + 单样本钉住、workspace 块 _raw_frames 传播、executor 友好错误映射。

Comment thread src/pt_snap_cli/query/executor.py Outdated
@staticmethod
def _execution_error(error: sqlite3.OperationalError) -> QueryExecutionError:
detail = str(error)
if "no such table: callstack" in detail or "no such column: callstackId" in detail:

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[低] 子串匹配对限定列名(qualified column)脆弱

"no such column: callstackId" 无法匹配 no such column: t.callstackId 或 no such column: trace_entry_0.callstackId(t. 直接跟在冒号空格后,子串不成立)。

当前惰性(裁判实测):真实 v1 库上三个调用栈模板都先触发缺表错误 no such table: callstack 而命中友好提示——包括 active_memory_callstack_at_event(其 ON t.callstackId = cs.id 的缺列错误永远后于 LEFT JOIN callstack 的缺表错误)。但一次模板改写(把裸列名改成限定引用)或一个「有 callstack 表但缺 callstackId 列」的混合布局库,即可让它静默退化为裸错误。

廉价加固:改用正则 re.search(r"no such column: (?:[\w.]+\.)?callstackId", detail),或直接匹配 "callstackId" in detail。

配套测试缺口:test_legacy_callstack_schema_error_is_actionable 只用合成 SQL SELECT callstack FROM callstack 锁定了缺表分支;callstack_analysis 在 v1 库上实际命中的缺列分支(no such column: callstackId)与 count() 路径均未测。

"""Assign one integer id per distinct callstack text.

PyTorch snapshots repeat a small number of callstacks across a very large
number of events, and the pickle memo already shares the frame containers

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[提示] 此句与实测不符,且与下方已改写的段落自相矛盾

「the pickle memo already shares the frame containers behind them」——上一轮实测(628k 脱敏 fixture):628,837 个事件产生约 421,115 个不同外层容器,pickle memo 共享的是单个帧字典(因此 _by_frames 命中率约 99.3%),而非外层容器(这正是 _by_container 仅命中约 33% 的原因)。本段下方的查找段落已改写为按帧身份查找,此句宜一并修正(例如改为「the pickle memo already shares the individual frame objects behind them」)。

assert interner.records() == [{"id": 0, "callstack": ""}]


def test_distinct_frame_identities_do_not_produce_a_false_hit():

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[提示] 替代测试未覆盖样本钉住(exemplar pinning)这一安全机制本身

旧的 test_reclaimed_container_address_does_not_produce_a_false_hit 被删除后,id 复用隐患(本类唯一的安全不变量:「_by_frames 值中的样本容器必须钉住键中全部帧 dict」)失去了直接测试——新测试只验证两个存活对象不混叠([INNER]/[OUTER]),即使删掉 intern() 里的 (frames, callstack_id) 样本存储它也会通过。实现本身构造上正确(裁判核实),但建议用 weakref 钉住该不变量:例如 intern 一个事件后 weakref.ref(frames),intern 若干其他事件后断言 ref() 仍非 None(样本未释放)。

HAVING alloc_count >= {{ min_count }}
AND total_size >= {{ min_size }}
) AS grouped
LEFT JOIN callstack c ON c.id = grouped.callstackId

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[提示] 悬空分组经结果映射后显示为字符串 "None"

LEFT JOIN 修复后悬空 callstackId 分组以 c.callstack = NULL 呈现,但经 ResultMapper(mapper.py:69,str 转换器 str(None))后会显示为字符串 "None"。兄弟模板 active_memory_callstack_at_event 用 COALESCE(cs.callstack, '[missing callstack]')。第一方导入库不可达此场景(覆盖性有测试保证),仅为一贸性对齐建议:COALESCE(c.callstack, '[missing callstack]') AS callstack。

@codeasier
codeasier merged commit 9e1095a into main Sep 6, 2026
6 checks passed
@codeasier
codeasier deleted the perf/import-callstack-dedup branch October 3, 2026 19:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant