diff --git a/docs/en/querying.md b/docs/en/querying.md index 6d363a5..62b0e37 100644 --- a/docs/en/querying.md +++ b/docs/en/querying.md @@ -102,7 +102,7 @@ across devices and there is no shared cross-device event order. | `preexisting_live` | Owner-event preexisting occupancy, not a leak count | | `block` | Merge proved allocation-event lifetimes and stable negative tokens once, then apply filters/order/page; latest containing-shard state is an observation, not state at arbitrary E | | `freed_block_lifetime` | Deduplicated captured allocations with a proved `free_completed` source; event-ID distance buckets, not time | -| `leak_detection` | Captured dynamic lifetimes still present without completion in the **terminal dataset shard**, not union of per-shard unfreed lists; survival is only a candidate, not a confirmed leak; `--slice` is rejected | +| `leak_detection` | Proved captured dynamic lifetimes observed as active/pending-free (`state IN (0,1)`) without completion in the **terminal dataset shard**, not union of per-shard unfreed lists; survival is only a candidate, not a confirmed leak; `--slice` is rejected | | `callstack_analysis` | All real stack-bearing trace actions, including free/segment/workspace, merged by full canonical source identity before global thresholds/ranking; weighted `avg_size=total_size/alloc_count` | | Runtime overrides/custom templates/arbitrary SQL | Explicitly unsupported on datasets, even when named after a built-in; pass an explicit standalone member to use local SQL | @@ -146,6 +146,38 @@ involved `slice_indices`, device, fingerprint and boundary exclusion. Source cov is separate. Unproved historical lifecycle identities are shard-local, explicitly marked, never guessed by address/local stack ID and never asserted as proved leaks. +### Leak candidates and lifecycle identity + +`leak_detection` retains an intentional scope difference at semantics version **1**: + +- **Standalone SQL** selects `allocEventId >= 0` with a NULL/negative `freeEventId` + (and the size filter). It does not filter stored `state` or verify the referenced + allocation action. An explicit dataset member is queried with this local contract. +- **Dataset scope** requires a proved real allocation action from its owning shard, + a terminal-shard observation with `state=1` (`active_allocated`) or `state=0` + (`active_pending_free`), and a NULL/negative completion reference after merging + any proved completion. Inactive (`state=-1`) or earlier-only observations are + excluded even when `freeEventId=-1`. A free request is not a completed free; + absence of a completion does not override the state or allocation-proof checks. + +Thus an inactive block with a proved allocation and unknown free can appear in +standalone results but not dataset results. This is a schema-valid synthetic +counterexample, not evidence that real replay generates it or that replay is +defective. ATTACH, private scratch reduction and bounded Python fallback implement +the same dataset contract. Neither scope confirms a leak. + +Dataset `block` identity is separate from terminal survival. A proved allocation +event merges its observations across shards; `allocEventId=-1` with a negative +block ID uses a stable preexisting token, not allocation proof. Negative boundary +events never provide allocation proof. Other unproved observations retain +`unproved:slice::block:` identities even if block ID, address and +allocation reference collide across shards. They remain visible in `block`, with +`identity_status=unproved_shard_local` and missing allocation source, but are +excluded from dataset leak candidates. Merging them requires new cross-shard +identity evidence; equal addresses or unproved references alone are insufficient. +`source_coverage.unproved_identities` counts these shard-local identities before +filters/pages, independently of `has_more`/`truncated`. + Dataset reads detach batches before foreign-owner lookups, using the borrowed bounded Context LRU. One diminishing deadline includes all shards, sources, grouping and exact totals; reports explicitly share **one** budget across peak selection and attribution. Cumulative diff --git a/docs/zh/querying.md b/docs/zh/querying.md index 78ced13..8ee3a58 100644 --- a/docs/zh/querying.md +++ b/docs/zh/querying.md @@ -92,7 +92,7 @@ pt-snap query '' --device 0 --template-use event --params '{"id":123}' | `preexisting_live` | 所有者事件的 preexisting 占用,不是泄漏数 | | `block` | 已证明的分配事件生命周期及稳定负 token 去重后筛选/排序/分页;最新出现片 state 是观察,不是任意 E 的状态 | | `freed_block_lifetime` | 已采集分配与已证明 `free_completed` 来源的去重生命周期;event-ID 距离桶,不是时间 | -| `leak_detection` | **数据集终片**中仍存在且无完成释放的已采集动态生命周期,不拼接逐片未释放列表;存活仅为候选,不证明泄漏;拒绝 `--slice` | +| `leak_detection` | **数据集终片**中观察为活跃/待释放(`state IN (0,1)`)且无完成释放、分配已证明的已采集动态生命周期,不拼接逐片未释放列表;存活仅为候选,不证明泄漏;拒绝 `--slice` | | `callstack_analysis` | 全部带栈真实 trace action(含 free/segment/workspace),完整 canonical 来源身份合并后再做全局阈值/排名;加权 `avg_size=total_size/alloc_count` | | runtime override/自定义模板/任意 SQL | 数据集明确不支持,即使命名为内建模板;需显式传单库 member 使用局部 SQL | @@ -127,6 +127,33 @@ exactness 标记诚实。`has_more/truncated` 描述窗口,不表示来源/范 请求区间、适用时实际真实事件覆盖、涉及 `slice_indices`、设备、指纹及边界排除;来源覆盖 独立。未证明历史身份按片局部单独表达,不按地址/局部栈 ID 猜合并,不断言为已证明泄漏。 +### 泄漏候选与生命周期身份 + +`leak_detection` 在语义版本 **1** 中保留有意的作用域差异: + +- **单库 SQL** 选择 `allocEventId >= 0` 且 `freeEventId` 为 NULL/负数的行 + (并应用大小筛选),不按存储的 `state` 筛选,也不验证引用的分配 action。 + 显式查询数据集成员文件时采用此局部合同。 +- **数据集作用域** 要求在所属片证明真实 allocation action,终片观察为 + `state=1`(`active_allocated`)或 `state=0`(`active_pending_free`),且归并 + 已证明的完成释放后,完成释放引用仍为 NULL/负数。即使 `freeEventId=-1`, + inactive(`state=-1`)或仅较早分片出现的观察也排除。释放请求不等于完成释放; + 缺少完成释放不能取代 state 或分配来源证明检查。 + +因此,分配已证明、释放未知的 inactive block 可出现在单库结果中,却不出现在数据集 +结果中。这是符合 schema 的合成差分反例,不证明真实 replay 会生成此组合,也不证明 +replay 存在缺陷。ATTACH、私有 scratch 归约和受预算约束的 Python fallback 实现同一 +数据集合同。两种作用域都不确认泄漏。 + +数据集 `block` 身份与终片存活相互独立。已证明的分配事件跨片归并其观察; +`allocEventId=-1` 且 block ID 为负数时采用稳定的 preexisting token,但不构成分配证明。 +负边界事件从不提供分配证明。其他未证明观察保留 +`unproved:slice::block:` 身份,即使跨片 block ID、地址和分配引用相同也不归并。 +这些观察仍可在 `block` 中看到,标记为 `identity_status=unproved_shard_local`、分配来源 +缺失,但不进入数据集泄漏候选。归并需要新的跨片身份证据;仅地址或未证明引用相同不足以 +支持归并。`source_coverage.unproved_identities` 在筛选/分页前统计这些按片身份,与 +`has_more`/`truncated` 独立。 + 数据集读取先分离批次再查外片来源,使用借用的有界 Context LRU。所有片、来源、分组、 精确总数共用递减 deadline;报告的峰值选择与归因明确共用**一个**预算。累计取回/输出工作默认上限为 **100000 行 / 64 MiB 序列化值**,含重复来源读取;超限整体失败,不返回冒充完整的部分 diff --git a/src/pt_snap_cli/query/templates/basic/block.yaml b/src/pt_snap_cli/query/templates/basic/block.yaml index 195ed74..c2f706b 100644 --- a/src/pt_snap_cli/query/templates/basic/block.yaml +++ b/src/pt_snap_cli/query/templates/basic/block.yaml @@ -125,7 +125,7 @@ queries: units: text metric_semantics: identifier scope: captured_range - interpretation_limits: ["Dataset only, optional: generation/device/proved allocation-event or stable negative lifetime token; unproved histories use a separate shard-local identity. Never address or local stack identity."] + interpretation_limits: ["Dataset only, optional: generation/device/proved allocation-event or stable negative lifetime token; unproved histories retain separate unproved:slice::block: identities even when block IDs, addresses and unproved allocation references coincide across shards. Never address or local stack identity."] - column: identity_status type: str units: text diff --git a/src/pt_snap_cli/query/templates/business/leak_detection.yaml b/src/pt_snap_cli/query/templates/business/leak_detection.yaml index 374b1bf..5b70734 100644 --- a/src/pt_snap_cli/query/templates/business/leak_detection.yaml +++ b/src/pt_snap_cli/query/templates/business/leak_detection.yaml @@ -20,6 +20,8 @@ queries: semantics_version: 1 interpretation_limits: - "Rows are candidates still without a free-completion event in the captured range; end-of-trace survival is not a confirmed leak." + - "Standalone SQL checks allocEventId >= 0 and a NULL/negative freeEventId, without state filtering or allocation-action proof." + - "Dataset scope additionally requires a proved real allocation and a terminal-shard active_allocated/active_pending_free observation (state IN (0,1)); inactive plus unknown-free and earlier-only observations are excluded. ATTACH, scratch and bounded fallback share this contract." - "Event IDs and ID distances are ordering markers, not elapsed time." query: | SELECT id, address, size, allocEventId diff --git a/tests/core/test_dataset_lifecycle_contract.py b/tests/core/test_dataset_lifecycle_contract.py new file mode 100644 index 0000000..3e064f6 --- /dev/null +++ b/tests/core/test_dataset_lifecycle_contract.py @@ -0,0 +1,255 @@ +"""Synthetic schema-valid contracts, not evidence of an upstream replay defect.""" + +import copy +import sqlite3 +from contextlib import closing +from pathlib import Path + +import pytest + +from pt_snap_cli.api import SnapshotAnalyzer +from pt_snap_cli.core import dataset_global +from pt_snap_cli.core import dataset_lifecycle_scratch as scratch +from pt_snap_cli.core import dataset_lifecycle_sql as sql +from pt_snap_cli.core.dataset_contract import validate_dataset +from pt_snap_cli.core.query_limits import QueryLimits +from tests.test_dataset_focus import hashes, make_dataset + + +@pytest.fixture(autouse=True) +def isolate(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + monkeypatch.setenv("HOME", str(tmp_path)) + monkeypatch.setattr(Path, "home", lambda: tmp_path) + monkeypatch.delenv("PT_SNAP_DB_PATH", raising=False) + monkeypatch.delenv("PT_SNAP_QUERY_TIMEOUT", raising=False) + + +def standalone(root, target): + """Copy all real events and latest block observations into local SQL tables.""" + members = sorted(root.glob("device_0/*.db")) + with closing(sqlite3.connect(target)) as out: + with closing(sqlite3.connect(members[0])) as first: + first.backup(out) + with out: + out.execute("DELETE FROM trace_entry_0 WHERE id<0") + for path in members[1:]: + with closing(sqlite3.connect(path)) as source: + out.executemany( + "INSERT INTO trace_entry_0 VALUES (?,?,?,?,?,?,?,?,?)", + source.execute("SELECT * FROM trace_entry_0 WHERE id>=0").fetchall(), + ) + out.executemany( + "INSERT OR REPLACE INTO block_0 VALUES (?,?,?,?,?,?,?)", + source.execute("SELECT * FROM block_0").fetchall(), + ) + return target + + +def dataset_query(root, monkeypatch, route, template, params=None): + """Force a real reducer on tiny inputs and prove it executed, without faking rows.""" + called = [] + original_attach = sql._AttachedSources.__enter__ + original_build = scratch._ScratchSources.build + original_fallback = dataset_global._lifecycles + + def attached(database): + result = original_attach(database) + assert database.connection.execute("PRAGMA query_only").fetchone()[0] == 1 + assert {row[1] for row in database.connection.execute("PRAGMA database_list")} == { + "main", + "s0", + "s1", + } + called.append("attached") + return result + + def built(database, items, maximum): + result = original_build(database, items, maximum) + assert database.connection.execute("SELECT COUNT(*) FROM lifecycle_state").fetchone()[0] + called.append("scratch") + return result + + def fallback(*args): + result = original_fallback(*args) + called.append("fallback") + return result + + with monkeypatch.context() as patch: + # Zero ATTACH capacity selects the existing bounded Python path. Lowering + # the observation threshold selects scratch without a huge fixture. + patch.setattr(sql, "_attachment_limit", lambda: 0 if route == "fallback" else 2) + patch.setattr(sql, "_ATTACHED_OBSERVATION_LIMIT", 0 if route == "scratch" else 4096) + patch.setattr(sql._AttachedSources, "__enter__", attached) + patch.setattr(scratch._ScratchSources, "build", built) + patch.setattr(dataset_global, "_lifecycles", fallback) + with SnapshotAnalyzer(root, query_limits=QueryLimits()) as analyzer: + result = analyzer.execute_query(template, params, exact_total=True) + assert called == [route] + return result + + +def equivalent_dataset_results(results): + normalized = [] + for result in results: + value = copy.deepcopy(result) + # Different execution plans legitimately perform different source reads. + count = value["scope"]["source_coverage"].pop("source_queries") + assert type(count) is int and count > 0 + normalized.append(value) + assert normalized[1:] == [normalized[0]] * (len(normalized) - 1) + + +@pytest.mark.parametrize("state", [-1, 0, 1], ids=["inactive", "pending_free", "active"]) +@pytest.mark.parametrize("free", [-1, 3], ids=["unknown_free", "completed_free"]) +@pytest.mark.parametrize("terminal", [True, False], ids=["terminal", "earlier_only"]) +def test_leak_state_contract_across_standalone_attached_scratch_and_fallback( + tmp_path, monkeypatch, state, free, terminal +): + root = make_dataset(tmp_path / "states", devices=(0,), slices=2) + for index in range(2): + with closing(sqlite3.connect(root / f"device_0/slice_{index:05d}.db")) as conn, conn: + conn.execute("UPDATE trace_entry_0 SET action=5 WHERE id=1") + conn.execute("UPDATE trace_entry_0 SET action=6 WHERE id=3") + if index == 0 or terminal: + conn.execute("INSERT INTO block_0 VALUES (0,100,8,8,?,0,?)", (state, free)) + # In particular, inactive + unknown-free is admitted by the format. This + # crafted table combination makes no claim about what real replay produces. + validate_dataset(root) + single = standalone(root, tmp_path / "standalone.db") + before = hashes(root) + with SnapshotAnalyzer(single, query_limits=QueryLimits()) as analyzer: + local = analyzer.execute_query("leak_detection", exact_total=True) + assert [row["id"] for row in local["rows"]] == ([0] if free == -1 else []) + assert all(set(row) == {"id", "address", "size", "allocEventId"} for row in local["rows"]) + results = [ + dataset_query(root, monkeypatch, route, "leak_detection") + for route in ("attached", "scratch", "fallback") + ] + equivalent_dataset_results(results) + expected = terminal and state in (0, 1) and free == -1 + for result in results: + assert [row["id"] for row in result["rows"]] == ([0] if expected else []) + assert result["total"] == int(expected) and result["total_is_exact"] + assert not result["has_more"] and not result["truncated"] + assert result["semantics_version"] == local["semantics_version"] == 1 + scope = result["scope"] + assert scope["slice_indices"] == [0, 1] and scope["range_complete"] + assert scope["boundary_events_included"] is False + coverage = scope["source_coverage"] + assert coverage["lifecycles"] == coverage["allocation_events_resolved"] == 1 + assert coverage["free_events_resolved"] == int(free == 3) + assert coverage["allocation_source_complete"] and coverage["unproved_identities"] == 0 + for row in result["rows"]: + assert row["terminal_survivor"] and row["observation_slice_index"] == 1 + assert row["state"] == state and row["state_scope"] == "latest_slice_observation" + assert row["identity_status"] == "proved_allocation_event" + assert row["allocation_source"]["slice_index"] == 0 + assert row["allocation_source"]["event"]["action"] == 4 + assert row["free_source"] is None and row["free_source_status"] == "live_or_unknown" + assert hashes(root) == before + + +@pytest.mark.parametrize( + "boundary_action", [0, 2], ids=["segment_boundary", "segment_alloc_boundary"] +) +def test_unproved_collisions_stay_shard_local_but_proved_and_negative_tokens_merge( + tmp_path, monkeypatch, boundary_action +): + root = make_dataset(tmp_path / "identities", devices=(0,), slices=2) + for index in range(2): + with closing(sqlite3.connect(root / f"device_0/slice_{index:05d}.db")) as conn, conn: + conn.execute("UPDATE trace_entry_0 SET action=5 WHERE id=1") + conn.execute("UPDATE trace_entry_0 SET action=6 WHERE id=3") + conn.execute("UPDATE trace_entry_0 SET action=? WHERE id=-1", (boundary_action,)) + # Same local ID, address and unproved allocation reference in both + # shards must not become one proved lifecycle, even with equal bytes. + conn.executemany( + "INSERT INTO block_0 VALUES (?,?,?,?,?,?,?)", + [ + (0, 100, 8, 8, -1, 0, 3), + (1, 100, 8, 8, 1, 1, -1), + (-1, 100, 8, 8, 1, -1, -1), + (-2, 100, 8, 8, 1, -1, -1), + ], + ) + if index == 1: + conn.execute("INSERT INTO block_0 VALUES (2,100,8,8,0,2,-1)") + assert conn.execute("SELECT action FROM trace_entry_0 WHERE id=-1").fetchone() == ( + boundary_action, + ) + validate_dataset(root) + single = standalone(root, tmp_path / "standalone.db") + before = hashes(root) + by_template = {} + for template in ("block", "leak_detection", "freed_block_lifetime"): + results = [ + dataset_query(root, monkeypatch, route, template) + for route in ("attached", "scratch", "fallback") + ] + equivalent_dataset_results(results) + by_template[template] = results[0] + coverage = results[0]["scope"]["source_coverage"] + assert coverage["coverage_scope"] == "all_deduplicated_lifecycles_before_filters_and_page" + assert coverage["lifecycles"] == 6 + assert coverage["allocation_events_resolved"] == 2 + assert coverage["free_events_resolved"] == 1 + assert coverage["unproved_identities"] == 2 + assert coverage["allocation_source_complete"] is False + rows = by_template["block"]["rows"] + unproved = [row for row in rows if row["id"] == 1] + assert len(unproved) == 2 + for row in unproved: + index = row["observation_slice_index"] + assert row["lifecycle_id"].endswith(f":unproved:slice:{index}:block:1") + assert row["identity_status"] == "unproved_shard_local" + assert row["allocation_source"] is None and row["allocation_source_status"] == "missing" + assert row["terminal_survivor"] == (index == 1) + assert {row["observation_slice_index"] for row in unproved} == {0, 1} + for row in rows: + if row["id"] < 0: + # A negative boundary event with the same numeric ID supplies no + # allocation proof. The stable preexisting token remains separate. + assert row["identity_status"] == "stable_negative_token" + assert row["lifecycle_id"].endswith(f":preexisting:{row['id']}") + assert row["allocation_source"] is None + assert row["allocation_source_status"] == "unknown_preexisting" + assert len({row["lifecycle_id"] for row in rows}) == 6 + assert [row["id"] for row in by_template["leak_detection"]["rows"]] == [2] + assert by_template["freed_block_lifetime"]["rows"] == [ + {"lifetime_events": "<1k", "block_count": 1, "size_bytes": 8} + ] + with SnapshotAnalyzer(single, query_limits=QueryLimits()) as analyzer: + # Local SQL checks the stored reference sign, not cross-shard source + # proof: the unproved action=5 reference deliberately remains visible. + local = analyzer.execute_query("leak_detection", exact_total=True) + assert [row["id"] for row in local["rows"]] == [2, 1] + assert analyzer.execute_query("block", exact_total=True)["total"] == 5 + assert hashes(root) == before + + +def test_public_metadata_explains_existing_scope_contract_without_version_change(): + with SnapshotAnalyzer(query_limits=QueryLimits()) as analyzer: + leak = analyzer.get_template_info("leak_detection") + block = analyzer.get_template_info("block") + catalog = {entry["name"]: entry for entry in analyzer.list_capabilities()["templates"]} + assert leak["semantics_version"] == 1 + assert leak["dataset_support"]["semantics"] == ( + "terminal_dataset_dynamic_survivors_not_confirmed_leaks" + ) + limits = " ".join(leak["interpretation_limits"]) + assert ( + "Standalone SQL" in limits + and "without state filtering or allocation-action proof" in limits + ) + assert "state IN (0,1)" in limits and "inactive plus unknown-free" in limits + lifecycle = next(field for field in block["output_schema"] if field["column"] == "lifecycle_id") + assert "unproved:slice::block:" in " ".join(lifecycle["interpretation_limits"]) + for name, info in (("leak_detection", leak), ("block", block)): + for field in ( + "semantics_version", + "interpretation_limits", + "output_schema", + "dataset_support", + ): + assert catalog[name][field] == info[field]