Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 41 additions & 6 deletions docs/en/sharded-snapshotdb.md
Original file line number Diff line number Diff line change
Expand Up @@ -242,12 +242,47 @@ full-text stack-statistics versions are documented with the same contracts.
`core.dataset_resolver.DatasetResolver.inspect(path)` returns an immutable
`ResolvedDataset` (or `None` for a standalone DB). Its `paths(QueryScope(...))`
addresses all matching files even for a cross-slice range, but does not execute it.
It revalidates the finalized P0 contract on each call and hashes the manifest and
members; no unbounded manifest/connection cache is retained. Hashing uses bounded
chunks and is proportional to artifact size, not a performance claim. Context LRU
size defaults to four; dataset generation changes invalidate contexts on next
lookup even when size/mtime are unchanged. Analyzer-owned caches close on exit;
injected caches remain caller-owned. Validation opens/closes slices sequentially.
By default **every independent inspection** fully validates the finalized contract
and hashes the manifest/members. File metadata alone is not content-integrity proof:
unknown or coarse filesystems may expose unchanged `ctime_ns` after a write.
Each query, including each query step within a multi-step report, calls `inspect`
again. Reports do not currently pin one `ResolvedDataset` across their steps or
invoke `ResolvedDataset.require_unchanged()`; there is no additional report-level
content-hash guard.

`PT_SNAP_DATASET_CACHE=immutable` explicitly opts into a process-wide LRU of at most
16 validated generations, without open connections. Set it only when you control
finalized, immutable publication and trust your filesystem's kernel change-time
semantics. **Do not enable it for files that may be modified in place**, live writers,
unknown/coarse/network filesystems, or filesystem snapshots that may be rolled back.
This is a caller responsibility contract, not a filesystem capability detected by
the tool. Unset it to retain strong validation; unknown values also use strong mode.
Windows and platforms without a usable POSIX change token remain in strong mode.

With that explicit opt-in, new resolver/service instances can reuse validated
results. Every cache hit during inspection still rechecks canonical paths,
sidecars, native membership and each file's device, inode, size, mtime, **ctime**,
mode and link count. These conservative checks catch ordinary replacements and
in-place edits with restored mtime, but do not replace the immutable-file contract.
For a fixed shard count, an opted-in hit performs no row scans or whole-file hashes:
work is O(shards), independent of event/block counts. This metadata-only reuse also
applies to report query steps; a cache hit does not rehash content. Default inspections
and cache misses remain proportional to content. Explicit `validate_dataset` /
`validate_native_dataset` calls always validate fully. Native admission verifies
hashes before and after validation and reuses the verified hashes for its fingerprint,
avoiding a third member-hash pass.

The query's shared deadline reaches admission: hashing checks every 1 MiB chunk,
Python validation loops check every 256 rows, and SQLite validation (including
`quick_check`) uses an interruptible progress handler, cleared on every exit.
Cancellation raises `QUERY_TIMEOUT`; no partial validation is cached. Device tab
completion reads only a size-bounded manifest (up to 1 MiB), never member SQLite
contents or hashes. Its candidates are hints, not proof the dataset is valid.

Context LRU size defaults to four; dataset content-generation changes invalidate
contexts on next lookup even when size/mtime are unchanged. Analyzer-owned caches
close on exit; injected caches remain caller-owned. Validation opens/closes slices
sequentially.

Building, missing/inconsistent slices, unknown base versions and unsafe paths
are rejected before focus writes or query execution. Before **any** SQLite open,
Expand Down
33 changes: 29 additions & 4 deletions docs/zh/sharded-snapshotdb.md
Original file line number Diff line number Diff line change
Expand Up @@ -198,10 +198,35 @@ selector。输出 `scope` 标识实际 DB/device/slice、受限真实区间、

`core.dataset_resolver.DatasetResolver.inspect(path)` 返回不可变 `ResolvedDataset`
(单库返回 `None`);其 `paths(QueryScope(...))` 可寻址跨片范围的全部文件,但不执行查询。
每次调用重新校验最终化 P0 合同并对 manifest/所有成员取内容哈希,不保存无界 manifest/
连接缓存。哈希按有界块读取,工作量与产物大小成正比,不宣称性能收益。Context LRU 默认
最多四个;数据集 generation 变化在下次 lookup 使旧 Context 失效,即使 size/mtime 相同。
analyzer 自有缓存在退出时关闭,注入缓存仍由调用者所有;校验逐片打开并关闭连接。
默认**每次独立 inspect 都完整校验**最终化合同,并对 manifest/成员取内容哈希。文件元数据
本身不是内容完整性证明:未知或低精度文件系统可能在写入后仍返回相同 `ctime_ns`。
每次查询(包括多步报告中的每个查询步骤)都会再次调用 `inspect`。报告目前不会跨步骤
固定同一个 `ResolvedDataset`,也不会调用 `ResolvedDataset.require_unchanged()`;
没有额外的报告级内容哈希守卫。

`PT_SNAP_DATASET_CACHE=immutable` 显式启用最多 16 个已校验 generation 的进程级 LRU,
不保留连接。仅在你控制最终化、不可变发布,且信任文件系统内核 change-time 语义时启用。
**可能被原地修改的文件、live writer、未知/低精度/网络文件系统或可能回滚的文件系统快照
不得启用**。这是调用方责任合同,不代表工具已检测文件系统能力。不设置该变量即保留强校验,
未知取值也使用强校验;Windows 或没有可用 POSIX change token 的平台始终使用强校验。

显式启用后,新建 resolver/service 可复用已校验结果。每次 inspect 缓存命中仍验证
规范路径、sidecar、原生成员集合,以及文件的设备号、inode、size、mtime、**ctime**、mode
和硬链接数。这些保守检查可检测常规替换及恢复 mtime 的原地修改,但不能代替不可变文件
合同。固定分片数时,显式启用后的命中不扫描行、不做整文件哈希,工作量为 O(分片数),
不随 event/block 数增长。报告中的查询步骤也适用这种仅检查元数据的复用;命中时不会
重新哈希内容。默认 inspect 及未命中仍与内容规模成正比。显式 `validate_dataset`
/ `validate_native_dataset` 始终完整校验。原生准入保留校验前后两遍哈希,fingerprint 直接
复用已验证摘要,不再做第三遍成员哈希。

查询共用 deadline 覆盖准入:哈希每个 1 MiB 块、Python 校验每 256 行检查一次;SQLite
校验(含 `quick_check`)使用可中断 progress handler,并在所有退出路径清除。
取消保留 `QUERY_TIMEOUT`,不会缓存不完整校验。设备 tab 补全只读取有界 manifest
(最多 1 MiB),不读取成员 SQLite 内容或整文件哈希;候选仅为提示,不证明数据集有效。

Context LRU 默认最多四个;数据集内容 generation 变化在下次 lookup 使旧 Context 失效,
即使 size/mtime 相同。analyzer 自有缓存在退出时关闭,注入缓存仍由调用者所有;
校验逐片打开并关闭连接。

building、缺片/矛盾片、未知基础版本、不安全路径在 focus 写入或查询前拒绝。
**任何** SQLite 打开之前,裸 P0 validator 与 resolver 均先检查**全部**成员/父目录别名及
Expand Down
6 changes: 3 additions & 3 deletions src/pt_snap_cli/completion.py
Original file line number Diff line number Diff line change
Expand Up @@ -55,8 +55,8 @@
if project_focus:
try:
with open(project_focus) as f:
data = json.load(f)

Check warning on line 58 in src/pt_snap_cli/completion.py

View workflow job for this annotation

GitHub Actions / Lint & Format

Type of "data" is Any (reportAny)
db_path = data.get(DB_PATH_KEY) or data.get("current_db_path")

Check warning on line 59 in src/pt_snap_cli/completion.py

View workflow job for this annotation

GitHub Actions / Lint & Format

Type of "db_path" is Any (reportAny)
if db_path:
return Path(db_path).expanduser()
except (json.JSONDecodeError, OSError):
Expand All @@ -77,9 +77,9 @@
try:
from pt_snap_cli.core.dataset_resolver import DatasetResolver

dataset = DatasetResolver().inspect(db_path)
if dataset is not None:
return [str(device) for device in dataset.device_ids]
devices = DatasetResolver().completion_device_ids(db_path)
if devices is not None:
return [str(device) for device in devices]
uri = f"file:{db_path}?mode=ro"
with closing(sqlite3.connect(uri, uri=True)) as conn:
cursor = conn.cursor()
Expand Down
49 changes: 36 additions & 13 deletions src/pt_snap_cli/core/dataset_contract.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,13 @@

from .dataset_files import require_readonly_member
from .errors import DatabaseSchemaError
from .validation_budget import (
ValidationBudget,
check_budget,
checked_rows,
sqlite_timeout,
validation_progress,
)

MANIFEST_SCHEMA_VERSION = 1
MSINSIGHT_REVISION = "101f65b877a267ffd5f66ea3834706057ba243e5"
Expand Down Expand Up @@ -118,12 +125,15 @@ def require_complete(self) -> None:
)


def parse_manifest(value: object, *, require_complete: bool = True) -> DatasetManifest:
def parse_manifest(
value: object, *, require_complete: bool = True, budget: ValidationBudget | None = None
) -> DatasetManifest:
"""Validate JSON-shaped data; inspection may explicitly accept building.

Unknown extension declarations are ignored, never promoted to capabilities.
This does not check files; use validate_dataset for finalized artifacts.
"""
check_budget(budget)
data = _object(value, "manifest")
version = _integer(data.get("schemaVersion"), "schemaVersion", maximum=2**31 - 1)
_need(
Expand All @@ -137,7 +147,7 @@ def parse_manifest(value: object, *, require_complete: bool = True) -> DatasetMa
devices_data = _object(data.get("devices"), "devices")
_need(bool(devices_data), "devices", "empty", "at least one event-bearing device required")
devices: list[DeviceRecord] = []
for key, value in devices_data.items():
for key, value in checked_rows(devices_data.items(), budget):
location = f"devices.{key}"
_need(
isinstance(key, str) and re.fullmatch(r"0|[1-9][0-9]*", key) is not None,
Expand All @@ -153,7 +163,9 @@ def parse_manifest(value: object, *, require_complete: bool = True) -> DatasetMa
)
ready = tuple(
_integer(item, location + ".readySlices", maximum=size - 1)
for item in _array(device.get("readySlices"), location + ".readySlices")
for item in checked_rows(
_array(device.get("readySlices"), location + ".readySlices"), budget
)
)
_need(
len(set(ready)) == len(ready),
Expand All @@ -170,7 +182,7 @@ def parse_manifest(value: object, *, require_complete: bool = True) -> DatasetMa
)
slices: list[SliceRecord] = []
expected_start = 0
for index, raw in enumerate(records):
for index, raw in enumerate(checked_rows(records, budget)):
loc = f"{location}.slices[{index}]"
item = _object(raw, loc)
number = _integer(item.get("index"), loc + ".index", maximum=2**31 - 1)
Expand Down Expand Up @@ -357,6 +369,7 @@ def _validate_slice(
device: DeviceRecord,
item: SliceRecord,
identities: dict[tuple[int, int], tuple[object, ...]],
budget: ValidationBudget | None = None,
) -> int:
loc = item.file
trace, block = f"trace_entry_{device.device_id}", f"block_{device.device_id}"
Expand Down Expand Up @@ -415,7 +428,7 @@ def _validate_slice(
).fetchone()
_need(bad is None, loc, "event_values", "invalid scalar type or negative byte metric")
boundary_count = conn.execute(f'SELECT COUNT(*) FROM "{trace}" WHERE id<0').fetchone()[0]
for row in conn.execute(f'SELECT * FROM "{block}"'):
for row in checked_rows(conn.execute(f'SELECT * FROM "{block}"'), budget):
block_id, address, size, requested, state, alloc, free = row
for name, value in zip(
("id", "address", "size", "requestedSize", "state", "allocEventId", "freeEventId"),
Expand Down Expand Up @@ -455,12 +468,15 @@ def _validate_slice(
return cast(int, boundary_count)


def validate_dataset(directory: Path | str) -> DatasetValidation:
def validate_dataset(
directory: Path | str, *, budget: ValidationBudget | None = None
) -> DatasetValidation:
"""Read a complete compatibility-v1 dataset without migration or pickle.

Checks are validation, not a filesystem sandbox or concurrent publication lock.
Missing pt_snap_metadata/extension tables are valid external base artifacts.
"""
check_budget(budget)
root = Path(directory).absolute()
_need(
root.resolve() == root and not root.is_symlink(),
Expand All @@ -472,11 +488,11 @@ def validate_dataset(directory: Path | str) -> DatasetValidation:
path = _safe_file(root, "manifest.json")
with path.open(encoding="utf-8") as source:
data: object = json.load(source, object_pairs_hook=_unique_object)
manifest = parse_manifest(data)
manifest = parse_manifest(data, budget=budget)
# Preflight ALL members before the first SQLite open, also for callers
# using this bare P0 validator rather than DatasetResolver.
for device in manifest.devices:
for item in device.slices:
for item in checked_rows(device.slices, budget):
member = _safe_file(root, item.file)
try:
require_readonly_member(root, member, immutable=True)
Expand All @@ -485,14 +501,21 @@ def validate_dataset(directory: Path | str) -> DatasetValidation:
identities: dict[tuple[int, int], tuple[object, ...]] = {}
boundaries = 0
for device in manifest.devices:
for item in device.slices:
for item in checked_rows(device.slices, budget):
db_path = _safe_file(root, item.file)
try:
with closing(
sqlite3.connect(db_path.as_uri() + "?mode=ro&immutable=1", uri=True)
) as conn:
with (
closing(
sqlite3.connect(
db_path.as_uri() + "?mode=ro&immutable=1",
uri=True,
timeout=sqlite_timeout(budget),
)
) as conn,
validation_progress(conn, budget),
):
conn.execute("PRAGMA query_only=ON")
boundaries += _validate_slice(conn, device, item, identities)
boundaries += _validate_slice(conn, device, item, identities, budget)
except sqlite3.DatabaseError as exc:
raise DatasetContractError(item.file, "sqlite", str(exc)) from exc
return DatasetValidation(
Expand Down
6 changes: 5 additions & 1 deletion src/pt_snap_cli/core/dataset_files.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
from pathlib import Path

from .errors import DatabaseSchemaError
from .validation_budget import ValidationBudget, check_budget


def require_readonly_member(root: Path, member: Path, *, immutable: bool = False) -> None:
Expand All @@ -27,9 +28,12 @@ def require_readonly_member(root: Path, member: Path, *, immutable: bool = False
raise DatabaseSchemaError(f"Dataset member uses persistent WAL mode: {member}")


def hash_file(path: Path) -> str:
def hash_file(path: Path, *, budget: ValidationBudget | None = None) -> str:
check_budget(budget)
digest = hashlib.sha256()
with path.open("rb") as source:
for chunk in iter(lambda: source.read(1024 * 1024), b""):
check_budget(budget)
digest.update(chunk)
check_budget(budget)
return digest.hexdigest()
Loading
Loading