Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions docs/en/querying.md
Original file line number Diff line number Diff line change
Expand Up @@ -153,10 +153,12 @@ fetched/output work is limited to **100000 rows / 64 MiB serialized values**, in
repeated source reads. Exceeding either fails the entire operation, never an alleged
complete partial result. Python materialization/sort are bounded by this work ceiling, **not**
by max_rows. This is not a process RSS limit: Python/SQLite, one batch and individual
cells have overhead. Strong manifest/member validation and hashing remain the default
and share that deadline: hash chunks, Python row batches and SQLite progress handlers
cells have overhead. Strong manifest/member admission remains the default; each
inspection hashes all content, reusing prior in-process row validation only when
content and generation match. Warm work remains O(bytes + shards), and new CLI
processes are cold. Admission shares that deadline: hash chunks, Python row batches and SQLite progress handlers
check cancellation. Individual filesystem calls are not OS-level hard-preempted.
`PT_SNAP_DATASET_CACHE=immutable` explicitly opts into process-local validation reuse
`PT_SNAP_DATASET_CACHE=immutable` explicitly opts into metadata-only validation reuse
only when the caller guarantees immutable finalized files and reliable filesystem
change times; see [dataset validation and cache contracts](sharded-snapshotdb.md).
No temporary merge DB, repair or artifact write.
Expand Down
66 changes: 40 additions & 26 deletions docs/en/sharded-snapshotdb.md
Original file line number Diff line number Diff line change
Expand Up @@ -242,32 +242,46 @@ full-text stack-statistics versions are documented with the same contracts.
`core.dataset_resolver.DatasetResolver.inspect(path)` returns an immutable
`ResolvedDataset` (or `None` for a standalone DB). Its `paths(QueryScope(...))`
addresses all matching files even for a cross-slice range, but does not execute it.
By default **every independent inspection** fully validates the finalized contract
and hashes the manifest/members. File metadata alone is not content-integrity proof:
unknown or coarse filesystems may expose unchanged `ctime_ns` after a write.
Pinned report-operation guards also rehash content by default, without repeating
row validation.

`PT_SNAP_DATASET_CACHE=immutable` explicitly opts into a process-wide LRU of at most
16 validated generations, without open connections. Set it only when you control
finalized, immutable publication and trust your filesystem's kernel change-time
semantics. **Do not enable it for files that may be modified in place**, live writers,
unknown/coarse/network filesystems, or filesystem snapshots that may be rolled back.
This is a caller responsibility contract, not a filesystem capability detected by
the tool. Unset it to retain strong validation; unknown values also use strong mode.
Windows and platforms without a usable POSIX change token remain in strong mode.

With that explicit opt-in, new resolver/service instances can reuse validated
results. Every hit and pinned-operation guard still rechecks canonical paths,
sidecars, native membership and each file's device, inode, size, mtime, **ctime**,
mode and link count. These conservative checks catch ordinary replacements and
in-place edits with restored mtime, but do not replace the immutable-file contract.
For a fixed shard count, an opted-in hit performs no row scans or whole-file hashes:
work is O(shards), independent of event/block counts. Default inspections and cache
misses remain proportional to content. Explicit `validate_dataset` /
`validate_native_dataset` calls always validate fully. Native admission verifies
hashes before and after validation and reuses the verified hashes for its fingerprint,
avoiding a third member-hash pass.
By default, the first inspection fully validates the finalized contract and hashes
its content before/after validation. A process-wide LRU holds at most 16 validated
generations, with no open connections or row-level validation collections. Each
subsequent inspection still hashes the manifest and **every member** and checks
canonical paths, sidecars, native membership and file metadata before/after hashing.
Only an unchanged generation with the same content fingerprint reuses its validated
result, avoiding repeated SQLite/schema/row validation. Changed metadata or content
falls back to complete validation; failures and cancellation evict the entry.

Default warm work is **O(bytes + shards)**, not O(1) or independent of dataset size.
It saves repeated row-validation CPU and temporary memory, not full-file reads.
Every new CLI process is cold; this is not a persistent or cross-process cache.
Compatibility-v1 cold admission adds a pre-validation member hash to its existing
post-validation hash; native admission retains two passes, reusing verified hashes
for the fingerprint rather than adding a third pass. Pinned report-operation guards
also rehash content by default, without repeating row validation. Explicit
`validate_dataset` / `validate_native_dataset` calls always validate fully.

File metadata alone is not content-integrity proof: unknown or coarse filesystems
may expose unchanged `ctime_ns` after a write. Unsetting `PT_SNAP_DATASET_CACHE`
selects the default content-verified cache. Setting it to `off` or an unknown value
fully revalidates on every inspection without caching; strong hashing remains active.

`PT_SNAP_DATASET_CACHE=immutable` explicitly opts into metadata-only reuse in the
same bounded process cache. Set it only when you control finalized, immutable
publication and trust your filesystem's kernel change-time semantics. **Do not
enable it for files that may be modified in place**, live writers, unknown/coarse/
network filesystems, or filesystem snapshots that may be rolled back. This is a
caller responsibility contract, not a filesystem capability detected by the tool.
Windows and platforms without a usable POSIX change token fully revalidate in this
mode. An immutable-mode compatibility admission cannot seed default reuse until
it has passed the stronger content-bracketed admission.

With that explicit opt-in, every hit and pinned-operation guard still rechecks
canonical paths, sidecars, native membership and each file's device, inode, size,
mtime, **ctime**, mode and link count. These conservative checks catch ordinary
replacements and in-place edits with restored mtime, but do not replace the
immutable-file contract. An opted-in hit performs no row scans or whole-file hashes:
work is O(shards), independent of event/block counts. None of these checks provides
a concurrent-writer lock or an atomic filesystem snapshot.

The query's shared deadline reaches admission: hashing checks every 1 MiB chunk,
Python validation loops check every 256 rows, and SQLite validation (including
Expand Down
7 changes: 4 additions & 3 deletions docs/zh/querying.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,10 +131,11 @@ exactness 标记诚实。`has_more/truncated` 描述窗口,不表示来源/范
精确总数共用递减 deadline;报告的峰值选择与归因明确共用**一个**预算。累计取回/输出工作上限为
**100000 行 / 64 MiB 序列化值**,含重复来源读取;超限整体失败,不返回冒充完整的部分
结论。物化/排序由工作上限约束,**不由 max_rows 约束**;不是进程 RSS 上限,Python/
SQLite、一个批次及单个 cell 有额外开销。默认仍对 manifest/成员执行完整校验和 hash,
并共用该 deadline:hash 分块、Python 行批次及 SQLite progress handler 都检查取消。
SQLite、一个批次及单个 cell 有额外开销。默认仍执行强准入:每次对 manifest/全部成员
取哈希,仅当内容及 generation 一致时复用进程内逐行校验结果。热路径仍为 O(字节数 +
分片数),新 CLI 进程仍冷启动。准入共用该 deadline:hash 分块、Python 行批次及 SQLite progress handler 都检查取消。
单个文件系统调用不提供 OS 级硬抢占。仅当调用方保证最终产物不可变且文件系统变更时间
可靠时,才可通过 `PT_SNAP_DATASET_CACHE=immutable` 显式启用进程内校验复用;参见
可靠时,才可通过 `PT_SNAP_DATASET_CACHE=immutable` 显式启用纯元数据校验复用;参见
[数据集校验与缓存合同](sharded-snapshotdb.md)。不生成合并临时 DB、不修复或写产物。

兼容导出器默认**每设备每片 500000 个真实事件**。该存储容量与 **100000 行 / 64 MiB
Expand Down
46 changes: 29 additions & 17 deletions docs/zh/sharded-snapshotdb.md
Original file line number Diff line number Diff line change
Expand Up @@ -198,23 +198,35 @@ selector。输出 `scope` 标识实际 DB/device/slice、受限真实区间、

`core.dataset_resolver.DatasetResolver.inspect(path)` 返回不可变 `ResolvedDataset`
(单库返回 `None`);其 `paths(QueryScope(...))` 可寻址跨片范围的全部文件,但不执行查询。
默认**每次独立 inspect 都完整校验**最终化合同,并对 manifest/成员取内容哈希。文件元数据
本身不是内容完整性证明:未知或低精度文件系统可能在写入后仍返回相同 `ctime_ns`。
已固定 generation 的报告操作默认也重新哈希内容,但不重复逐行校验。

`PT_SNAP_DATASET_CACHE=immutable` 显式启用最多 16 个已校验 generation 的进程级 LRU,
不保留连接。仅在你控制最终化、不可变发布,且信任文件系统内核 change-time 语义时启用。
**可能被原地修改的文件、live writer、未知/低精度/网络文件系统或可能回滚的文件系统快照
不得启用**。这是调用方责任合同,不代表工具已检测文件系统能力。不设置该变量即保留强校验,
未知取值也使用强校验;Windows 或没有可用 POSIX change token 的平台始终使用强校验。

显式启用后,新建 resolver/service 可复用已校验结果。每次命中及报告 generation 检查仍验证
规范路径、sidecar、原生成员集合,以及文件的设备号、inode、size、mtime、**ctime**、mode
和硬链接数。这些保守检查可检测常规替换及恢复 mtime 的原地修改,但不能代替不可变文件
合同。固定分片数时,显式启用后的命中不扫描行、不做整文件哈希,工作量为 O(分片数),
不随 event/block 数增长。默认 inspect 及未命中仍与内容规模成正比。显式 `validate_dataset`
/ `validate_native_dataset` 始终完整校验。原生准入保留校验前后两遍哈希,fingerprint 直接
复用已验证摘要,不再做第三遍成员哈希。
默认首次 inspect 完整校验最终化合同,并在校验前后对内容取哈希。进程级 LRU 最多保留
16 个已校验 generation,不保留连接或逐行校验集合。后续 inspect 仍对 manifest 和
**全部成员**取内容哈希,并在哈希前后检查规范路径、sidecar、原生成员集合和文件元数据。
只有 generation 未变且完整 fingerprint 相同才复用结果,跳过重复 SQLite/schema/逐行
校验。元数据或内容变化回退完整校验;失败或取消会清除对应缓存条目。

默认热路径仍是 **O(内容字节数 + 分片数)**,不是 O(1),也不独立于数据规模。节省的是
重复逐行校验 CPU 和临时内存,不是整文件读取。每个新 CLI 进程仍为冷启动;没有持久化
或跨进程缓存。兼容 v1 冷准入在已有校验后哈希外增加一遍校验前成员哈希;原生准入仍为
两遍,fingerprint 复用已验证摘要,不增加第三遍。已固定 generation 的报告操作默认仍
重新哈希内容,不重复逐行校验。显式 `validate_dataset` / `validate_native_dataset`
始终完整校验。

文件元数据本身不是内容完整性证明:未知或低精度文件系统可能在写入后仍返回相同
`ctime_ns`。不设置 `PT_SNAP_DATASET_CACHE` 使用默认内容验证缓存;设为 `off` 或未知
取值时每次完整重验且不缓存,仍执行强内容哈希。

`PT_SNAP_DATASET_CACHE=immutable` 显式选择同一有界进程缓存的纯元数据复用路径。
仅在你控制最终化、不可变发布,且信任文件系统内核 change-time 语义时启用。**可能被
原地修改的文件、live writer、未知/低精度/网络文件系统或可能回滚的文件系统快照不得
启用**。这是调用方责任合同,不代表工具已检测文件系统能力。Windows 或没有可用
POSIX change token 的平台在此模式下每次完整重验。immutable 模式准入的兼容数据必须
先通过更强的前后内容哈希准入,才能用于默认缓存命中。

显式启用后,每次命中及报告 generation 检查仍验证规范路径、sidecar、原生成员集合,
以及文件的设备号、inode、size、mtime、**ctime**、mode 和硬链接数。这些保守检查可检测
常规替换及恢复 mtime 的原地修改,但不能代替不可变文件合同。显式启用后的命中不扫描
行、不做整文件哈希,工作量为 O(分片数),不随 event/block 数增长。这些检查均不提供
并发 writer 锁或原子文件系统快照。

查询共用 deadline 覆盖准入:哈希每个 1 MiB 块、Python 校验每 256 行检查一次;SQLite
校验(含 `quick_check`)使用可中断 progress handler,并在所有退出路径清除。
Expand Down
Loading
Loading