Skip to content

fix: preserve compatible cache reuse after Insight derived tables - #267

Closed
codeasier wants to merge 2 commits into
mainfrom
fix/266-compatible-content-proof
Closed

codeasier wants to merge 2 commits into
mainfrom
fix/266-compatible-content-proof

Conversation

@codeasier

Copy link
Copy Markdown
Owner

Summary

Refs #266. This is a focused runtime fix with synthetic regressions, not a repeat of the completed full E2E acceptance or a release change. Keep the issue open for review.

  • Add version-1 ptSnap.memberContent evidence for every original SQLite schema object and every original table, including metadata, dictionary and lifecycle references. Type-tagged, length-framed row encoding preserves NULL/integer/real/text/blob distinctions and duplicate multiplicity; SQLite sorts and Python streams the rows.
  • Retain exact file hashes as the fast path. On stable byte drift, compare the saved base proof and permit only one newly added ordinary memory_allocation_cache_v1_<this member deviceId> table. Validate schema before hashing base rows; do not scan permitted derived cache content. Reject other table names/versions/devices, new indexes/views/triggers, and base schema/data changes.
  • Legacy exports without a prior proof still reuse only when exact bytes match. Changed legacy bytes report lost integrity evidence, without claiming base-data corruption. No proof backfill, cache repair, manifest rewrite or forced replacement.
  • Bracket inspection with manifest/member hashes, full file/directory generations, exact inventory and path/sidecar checks. Keep member metadata-to-identity validation. These guards detect observed changes; they do not provide a filesystem lock or support live writers.
  • Update bilingual docs and README entry points. Existing fixed-version GUI evidence remains historical evidence; no new GUI claim is made.

Validation

Frozen source tree: 1a2e414e91d73be9e9a0048bdf5af9009332a21b.

  • Related import/export/metadata/dataset/contracts/governance/package suites: 480 passed.
  • 52 new focused cases, including 14 mutation/TOCTOU cases with force off/on, unchanged focus and preserved target bytes.
  • Owning new + existing compatibility suites with branch coverage: 139 passed, 92.55% combined coverage of the two proof/export modules (above the configured 90% gate).
  • Whole-repository Ruff and Black checks passed; basedpyright 0 errors, 1784 informational warnings. git diff --check passed.
  • Independent design and implementation review approved the final logic; final 14 race regressions passed again.
  • Isolated wheel build passed, and source/wheel byte-content audit passed. Wheel SHA256: ad6fdd9a0732a8c6c5eb2886e95c3d4c5be1ca046f09518598023f2898f94a21. This is a development verification artifact, not a release.
  • Initial new-tests-only coverage selection passed 52 tests but missed the aggregate 90% coverage threshold; expanding to the owning existing compatibility suites reached 92.55%. The first no-isolation build lacked setuptools-scm; the normal isolated build then succeeded.

Remote CI is pending and will be checked for this exact head. No full E2E, live snapshot/session, paid model, user Mac, #264 modification, release workflow change, merge or issue closure.

Snapshot runtime provenance

Snapshot provenance decision: no-update
Snapshot provenance no-update reason: Changes are limited to core proof/export/metadata logic, synthetic tests and documentation; no files under src/pt_snap_cli/snapshot/ change.

Refs #266. Keep legacy byte proofs and no-replace publication; allow only versioned device-local derived cache additions with unchanged base content.

@codeasier codeasier left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

性能评估(head 4b93721,对比 main f940ff7)

结论:方向正确。复杂度是线性的,没有 O(n²),峰值内存也没变化。不过有一个持续性成本,建议合并前处理掉。

实测(单机,页缓存是热的,每项跑 3 次)

输入用 wheel_scale_acceptance.write_scale_source 生成:2,000,002 个事件,--events-per-slice 500000,切成 4 个满分片(每个约 137 MB)加 1 个小分片。

场景 main PR
导出耗时 / 峰值 RSS 50.6–56.0 s / 1302 MB 61.3–65.2 s / 1302 MB(约 +16%)
复用,字节没变 约 4.5 s 约 5.1–5.6 s(多读一遍文件做 hash)
复用,加了 memory_allocation_cache_v1_<dev>(每片约 150 万行) 约 4.8 s,报 member hash changed(预期行为) 15.2–16.4 s,复用成功
overview / metadata,漂移后 — 约 21 s(未漂移时约 10–11 s)
SQLite 临时文件峰值 4.3 MB 74.9 MB(≈ 单表编码后的大小,不随分片数增加)

单分片拆解(50 万行):

  • hash_file 0.18 s;member_content 2.72 s,其中 trace_entry_0 这张表占 1.86 s。
  • 带 UDF 但不排序时是 1.64 s,纯 SELECT * 是 0.44 s。
  • 所以瓶颈是逐行调用 Python UDF pt_snap_row,约 3.3 µs/行;ORDER BY 只占约 10%。

主要风险

  1. proof 不会刷新。 Insight 写入缓存表后,文件字节永久漂移,manifest 里的 hash 也不更新。所以之后每次 import 复用、overview、metadata(ImportMetadataService.inspect → inspect_owned_cache)都会重新做一遍全量内容比对。按线性外推(推断),1 亿事件、200 个分片时,每次调用要多出约 9 分钟;callstack 更深时还会更多。
  2. 导出时一定会算 proof,每个成员至少多一次全表扫描。另外,build_compatible_manifest 前后已经各做了一次 hash,finalize 阶段的 inspect_owned_cache 又会再做两次。

建议

  • 内容比对通过后,把新的字节 hash 记下来(比如写到 sidecar 里,不改 manifest),这样下次直接走快路径。如果不打算这么做,至少在文档里写明这个持续成本。
  • 有 INTEGER PK 或稳定 rowid 的大表,改成按 ORDER BY rowid 流式读取,在 Python 里分批编码后做 sha256,不再每行调一次 UDF。单表实测快约 2.3 倍(0.82 s 对 1.86 s),也不需要临时 B-tree。没有稳定 rowid 的小表可以继续用现在的排序方案。另外试过"每行 sha256 后做多重集加法聚合、不排序",实测 2.12 s,反而更慢,不推荐。
  • 导出时已经算出的 hash 直接传给 finalize 复用。快路径上的后置复核可以改成比对 stat(size/mtime/inode),不必再读一遍整个文件。
  • 可选:各成员的 proof 互相独立,可以按进程并行计算。

以上数字都是实测;hash 次数是读代码统计的,大规模估算和 rowid 方案的整体收益属于推断(rowid 方案只测了单张表)。本评审只是记录,不构成 approve。

Avoid duplicate P0 validation within metadata/overview and retain a bounded
in-process ownership proof cache guarded by complete content hashes.
Keep cold-process proof costs and all mutation/transport checks explicit.

Refs #266

Copy link
Copy Markdown
Owner Author

已在 a237940 处理这条性能评审中的重复验证,并补充冷进程成本说明。

确认调用链:普通查询的 DatasetResolver 不要求 ownership proof,但 overview → ImportMetadataService.inspect → inspect_owned_cache,以及 metadata,确实会触发。现在 metadata 复用同次操作已经完成的 P0 校验,overview 传递同一个 resolved dataset,避免二次基础验证。该结果必须同时匹配根路径、完整内容 fingerprint、manifest 和文件 generation;P0 结果本身不能充当 ownership 证据。

另加最多 16 个根目录的进程内 LRU,仅缓存已经完成全部 ownership 校验的结果。key 覆盖完整 manifest 字节及所有成员文件字节,因此包含 proof 版本/schema、identity、metadata 等上下文。命中前后仍读取完整 hash,检查 inventory、sidecar、路径和硬链接;结果深拷贝、并发锁保护,失败驱逐。没有写入 artifact/sidecar,也没有用 stat 替代内容验证。默认和 immutable 模式下,同大小且恢复 mtime 的基础数据篡改都会拒绝。

小型隔离复核(合成 100,000 事件、4 个分片,每片新增 25,000 行派生缓存,各 3 次;对比本 PR 上一 head 4b93721,不是 main 或本评审的 200 万事件测试):

  • 冷 metadata:1.080–1.192 s → 0.800–0.930 s;基础验证 2 次 → 1 次,逻辑 proof 仍为 4 次。
  • 冷 overview:1.082–1.133 s → 0.820–0.875 s;基础验证 2 次 → 1 次,完整 hash 调用 25 次 → 20 次。
  • 冷 import 仍约 0.78–0.85 s,仍需 4 次逻辑 proof;不主张这项获得实质改善。
  • 同进程 ownership 已预热:import 0.051–0.058 s;resolver 也预热后 metadata/overview 为 0.076–0.084 s,均不再扫描逻辑 proof。若 resolver 尚冷,两者首次仍约 0.31 s。这些不是独立 CLI 命令之间的收益。

持续成本的边界现已写入中英文文档:新的 CLI 进程在成员字节漂移后仍需扫描基础内容;首次导出生成 proof 的成本也仍然存在。本次没有引入跨进程持久化信任协议,也未改变 v1 的类型保真、多重集排序合同。按 rowid 改序会改变现有 proof 定义,不能直接作为等价优化替换。

新增 26 项缓存/复用回归;本地相关测试 699 passed / 2 skipped,Ruff、Black、类型检查(0 errors)及 wheel 构建/源码字节审计通过。独立安全 review 已检查缓存 key、跨目录隔离、并发、异常驱逐和 mutation guards。新 head 的远端 CI 正在运行,将按该 commit 核验终态。

Copy link
Copy Markdown
Owner Author

最终核验:当前 head a237940 的两组 CI 均已成功终止,无手动重跑。

  • Test:Python 3.10–3.13、Ruff/Black/basedpyright、Chromium 与 provenance 全部通过。Python 3.13 为 3000 passed / 14 skipped / 7 deselected,覆盖率 93.23%;wheel 构建、源码内容审计及 clean-wheel Agent 验收均通过。仅定时触发的 Installed-wheel dataset acceptance 按工作流条件跳过,不计作通过。
  • Platform compatibility:Windows 四个 non-slow 分片全部通过,合计覆盖本 PR 新增 78 项 proof/race/cache 回归;macOS 为 2984 passed / 14 skipped / 23 deselected,78 项新回归全部通过。

已读回分支与源码 tree bc8d7b1aff531ad4497aea6b58a694420c570f68,匹配本地验证内容;author/committer 均为本次批准的 codeasier noreply 身份。此条与上一条性能回复更新 PR 初始描述中的旧 head 验证记录。PR 保持 draft/open,#266 保持 open,等待审阅。

Copy link
Copy Markdown
Owner Author

关闭说明:#266 改为采用 #268 的 parserCache v1 方案。该方案与 msinsight !1305 共用同一份摘要契约,双向互用,并且允许 Insight 写入 parse.log、WAL/SHM。

本 PR 的严格规则(只允许新增 memory_allocation_cache_v1_<dev>)暂时不采用,原因有两点:

当前的要求是:数据异常(如删表、损坏)时必须明确报错,或者触发重建。具体意见见 #268 的评审。分支保留,其中的竞态测试思路后续可以参考。

@codeasier codeasier closed this Oct 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant