Skip to content

perf: skip Frame reconstruction during database import (#84) - #85

Merged
codeasier merged 3 commits into
mainfrom
fix/issue-#84-resolve
Aug 7, 2026
Merged

codeasier merged 3 commits into
mainfrom
fix/issue-#84-resolve

Conversation

@codeasier

@codeasier codeasier commented Aug 7, 2026 •

Copy link
Copy Markdown
Owner

📝 Description

针对高事件数、深调用栈场景优化数据库导入的对象重建与调用栈处理成本(issue #84)。

核心改动

  • 数据库导入路径改为保留原始 frame 字典(_raw_frames),不再为每个事件构建 Frame 对象;snapshot2db.dump() 通过内部 _raw_frames=True 显式选用该模式。
  • 公开的 DeviceSnapshot.from_dict() / TraceEntry.from_dict() / replay / split 仍保持 eager 构造,frames 契约与 mutation 语义不变。
  • raw frame 在构造期校验 filename / line / name,保持畸形输入原有失败时机。
  • 调用栈格式化统一为 reversed() + list comprehension,消除 frames[::-1]。
  • raw frames 完整贯穿事件 → block → segment → 合成段事件,保证最终数据库 callstack 与 eager 路径一致。
  • 直接 snapshot2db.dump() 使用同目录临时数据库并在成功后原子替换;失败不会暴露多设备部分结果,也不会破坏已有目标数据库。
  • 扩展 benchmarks/baseline_import.py:新增 frame 引用总数与平均/最大调用栈深度;使用 Python perf_counter() + resource.getrusage() 采集 wall/user/sys/RSS,支持 Linux 与 macOS(含 Darwin RSS 单位归一化),不再依赖 GNU/BSD /usr/bin/time。

🔗 Related Issue

Fixes #84

🧪 Testing

  • Tests added/updated(eager/fast 数据库逐行等价、调用栈顺序、缺失/空/不完整 frame、ID、序列化、segment 变更、失败原子清理、已有 DB 保留、跨平台 benchmark 指标解析)
  • Manual testing completed

全量测试:585 passed, 1 xfailed

基线测试(同一台机器、同一版 Python 标准库测量脚本,优化前 = base commit,优化后 = 本 PR)

样本 指标 优化前 优化后 变化
131k (131,275 events, 10.62M frames, avg/max depth 80.9/202) Wall 9.639s 3.879s -60%
Max RSS 1416.9MB 234.1MB -83%
DB 633.4MB 633.4MB 一致
628k (628,837 events, 42.01M frames, avg/max depth 66.8/182) Wall 42.820s 19.370s -55%
Max RSS 5617.3MB 707.9MB -87%
DB 5676.8MB 5676.8MB 一致

正确性:两样本 PRAGMA quick_check = ok,trace/block 行数、memory_peak / allocator_gap / block_summary 代表查询结果优化前后完全一致。

✅ Checklist

  • Code follows project guidelines
  • Documentation updated (if applicable)(PROVENANCE.md 追加记录)
  • No new warnings introduced
  • Changes tested locally

Snapshot Runtime Provenance

  • Snapshot provenance evaluation completed when snapshot runtime files change
    Snapshot provenance decision: updated
    Snapshot provenance no-update reason:

📋 Additional Notes

131k/628k LFS 样本已在本机拉取并用于同机优化前后实测。benchmark 的资源输出缺失或格式错误会显式失败,不再静默记录为 0。

codeasier added 3 commits August 7, 2026 17:32
Database import now keeps raw trace/segment/block frame dictionaries
instead of building a Frame object per frame reference, and formats
callstacks with reversed iterator and list comprehension. Public
snapshot replay and split keep eager Frame construction unchanged.

Baseline benchmark on 131k/628k sanitized samples (same machine,
before on dafaf4f, after on this change):

| sample | metric   | before | after | change |
|--------|----------|--------|-------|--------|
| 131k   | wall     | 9.600s | 3.510s | -63%   |
| 131k   | peak RSS | 1415.6MB | 233.0MB | -84%   |
| 628k   | wall     | 43.000s | 17.930s | -58%   |
| 628k   | peak RSS | 5615.8MB | 706.5MB | -87%   |

DB output is byte-for-byte identical (same row counts, callstacks,
representative queries, quick_check ok).
@codeasier

Copy link
Copy Markdown
Owner Author

已处理本轮评审意见:

  • benchmark 移除 GNU /usr/bin/time -f 依赖,改用父进程 perf_counter() 与子进程 resource.getrusage();Darwin ru_maxrss 从 bytes 归一化为 KiB,并补齐缺失/畸形指标测试。
  • raw frame 在构造期校验 filename/line/name,恢复 KeyError 时机。
  • snapshot2db.dump() 改为同目录临时 DB + os.replace() 原子发布,覆盖单设备、多设备部分提交、已有目标 DB 保留及缺失父目录场景。
  • 使用新版跨平台测量脚本重跑同机基线,PR 描述已更新:131k wall -60% / RSS -83%,628k wall -55% / RSS -87%。

本地验证:585 passed, 1 xfailed,Black/Ruff/diff check 全通过。

@codeasier
codeasier merged commit 2f96988 into main Aug 7, 2026
8 of 12 checks passed
@codeasier
codeasier deleted the fix/issue-#84-resolve branch August 7, 2026 15:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE] 优化高事件数深调用栈场景下的对象重建与调用栈处理

1 participant