Skip to content

perf: reuse content-verified dataset validation in process - #230

Merged
codeasier merged 1 commit into
mainfrom
fix/219-content-verified-cache
Oct 10, 2026
Merged

codeasier merged 1 commit into
mainfrom
fix/219-content-verified-cache

Conversation

@codeasier

Copy link
Copy Markdown
Owner

范围

Related to #219;本 PR 不关闭 #219。

默认同进程 dataset inspect 使用有界的 content-verified 验证结果缓存。每次命中前仍完整哈希 manifest/全部分片并核对路径、成员、sidecar 和前后 generation,只有与曾通过完整验证的 fingerprint 相同才省略重复 schema/quick_check/逐行验证。缓存不保留数据库连接或行级集合;返回副本,失败/取消/变更清除条目。

  • immutable 保持显式 opt-in;off/unknown 保留完整重验,模式切换不让弱缓存污染强内容缓存。
  • v1 首次建缓存增加验证前哈希;native 复用现有双哈希,不引入第三遍。
  • 显式 validate、报告阶段 strong hash guard、只读与deadline约束保留。
  • warm 仍 O(bytes + shards),新 CLI 进程仍 cold;不承诺免扫描、原子快照或默认完整性降级。

当前验证

最终 tree 793f37905ee809eba82ffe2acde3c9624a0f306a 已冻结。Ruff/Black、typecheck(0 errors)、build/wheel audit、clean-wheel acceptance 通过;最新非editable wheel中87项cache/immutable/report测试通过。独立复核较早同逻辑候选339项通过,并发现、验证修复同名manifest.json目录的取消清理键问题;最终微调仅显式丢弃5处pop返回值。

最终完整覆盖率与同树性能复测仍在运行,远端CI待运行,暂不宣称最终全套通过。性能记录会明确cold首次多一次hash成本、warm完整扫描字节量,以及同进程而非跨CLI的适用边界。

Copy link
Copy Markdown
Owner Author

最终冻结树 793f37905ee809eba82ffe2acde3c9624a0f306a(当前head56eb261b)的本地验收完成:

  • 全套2142 passed / 1 skipped,branch coverage92.78%,三项critical门槛通过;skipped的安装包默认规模用例已单独通过。
  • 同一非editable wheel:500002事件默认500000+2分片、producer/importer/exporter→CLI/API、跨边界释放/leak/report/只读哈希全部通过(35.82s);另87项安装后缓存/报告测试通过。
  • Ruff/Black/diff、build/wheel audit/clean-wheel通过;typecheck0 errors,1547既有warnings。
  • 最终树36cases/144calls性能复测:百万事件v1冷3.363s、热71.93ms;native冷3.658s、热60.63ms。off重复约2.922s/3.388s。热路径仍完整哈希全部成员字节,仅省validator和SQLite打开;v1首次额外哈希82,522,112字节,native仍两遍。

这些是同进程固定云端夹具的测量,不是所有机器的延时保证,也不优化新CLI进程冷启动。#219保留,未改变完整性/原子快照边界。当前exact-head远端CI另行核验。

Copy link
Copy Markdown
Owner Author

当前精确 head 56eb261 的完整 CI 已全部成功:https://github.com/codeasier/pt-snap-cli/actions/runs/38062097375 。Python3.10–3.13、lint/format/type和snapshot provenance全部通过。PR保持草稿,未合入;#219剩余边界如前述。

@codeasier
codeasier marked this pull request as ready for review October 10, 2026 15:52
@codeasier
codeasier merged commit bd74695 into main Oct 10, 2026
6 checks passed

Copy link
Copy Markdown
Owner Author

Merged by squash as bd74695 after final verification. The merged tree is exactly 793f37905ee809eba82ffe2acde3c9624a0f306a.

Final results supersede the earlier pending status in the PR body: full suite 2142 passed / 1 opt-in skip, 92.78% coverage; installed default-scale case passed separately on the same frozen wheel; lint, format, typecheck (0 errors), wheel audit and clean-wheel acceptance passed. Exact-head CI 38062097375 passed all six jobs. Main push CI: https://github.com/codeasier/pt-snap-cli/actions/runs/38065366192 (in progress).

This remains a partial improvement for #219; that issue stays open. Default warm checks still hash all bytes and shards, and new CLI processes remain cold.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant