Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
458 changes: 458 additions & 0 deletions .github/scripts/clean_wheel_agent_acceptance.py

Large diffs are not rendered by default.

21 changes: 5 additions & 16 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -114,22 +114,11 @@ jobs:
python -m pip install --upgrade build
python -m build --wheel

- name: Wheel smoke test (verify snapshot runtime is packaged)
- name: Clean-wheel Agent acceptance
if: matrix.python-version == '3.13'
run: |
set -euo pipefail
python -m pip install --force-reinstall dist/*.whl
# Confirm the first-party runtime was actually packaged into the wheel.
python -c "import pt_snap_cli.snapshot.tools.adaptors.snapshot2db as m; assert hasattr(m, 'run_dump_to_db'), 'snapshot run_dump_to_db missing in installed wheel'"
# End-to-end import against a real fixture — catches any packaging
# regression where setuptools drops src/pt_snap_cli/snapshot/.
FIXTURE=tests/fixtures/snapshots/snapshot_with_empty_cache.pkl
rm -rf /tmp/pt-snap-wheel-smoke
mkdir -p /tmp/pt-snap-wheel-smoke
DB=/tmp/pt-snap-wheel-smoke/snapshot_with_empty_cache.pkl.db
pt-snap import "$FIXTURE" --no-focus --output-dir /tmp/pt-snap-wheel-smoke
test -f "$DB"
SECOND_IMPORT=$(pt-snap import "$FIXTURE" --no-focus --output-dir /tmp/pt-snap-wheel-smoke)
echo "$SECOND_IMPORT" | grep -q '^Reused:'
pt-snap metadata "$DB" --json | python -c "import json, sys; data=json.load(sys.stdin); assert data['status'] == 'available'"
rm -rf /tmp/pt-snap-wheel-smoke
WHEEL=$(python -c 'from pathlib import Path; wheels = sorted(Path("dist").glob("*.whl")); assert len(wheels) == 1, wheels; print(wheels[0])')
python .github/scripts/clean_wheel_agent_acceptance.py \
--wheel "$WHEEL" \
--fixture tests/fixtures/snapshots/snapshot_with_empty_cache.pkl
4 changes: 3 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@

### 稳定性与工程

- Agent CLI 端到端评估补齐参数纠错、无设备数据库、真实竞争数据库、模板语义与 import focus 信封;post-change 契约任务成功率由 0 提升至 1,错误结论率由 0.8 降至 0,并约束调用与输出预算。
- CI 与 release gate 新增干净 wheel Agent 验收:在隔离 venv 中确认包与 helper 来自 wheel,并仅用结构化输出完成 helper 安装、trusted snapshot import、capabilities、overview 与 peak query。
- basedpyright 对 `src/pt_snap_cli/snapshot/` 与其余包路径同一套 error 门槛:补齐注解后移除 snapshot `executionEnvironments` 降级列表(#122 阶段 3)。

## [0.4.0] - Unreleased
Expand All @@ -34,7 +36,7 @@

### 新增

- 新增 `pt-snap-helper` 引导技能,并在 `pt-snap --help` 末尾加入 Agent 提示(优先使用已支持的 `--json`,用 `pt-snap skill list --json` 检查技能)。helper 只读路由,不自动安装包、导入 pickle 或改写 focus;随包分发走 `pt-snap skill`。
- 新增 `pt-snap-helper` 引导技能,并在 `pt-snap --help` 末尾加入 Agent 提示(优先使用已支持的 `--json`,无技能时可用 `pt-snap skill install pt-snap-helper --json` 安装并重启 Agent,用 `pt-snap skill list --json` 检查技能)。helper 只读路由,不自动安装包、导入 pickle 或改写 focus;随包分发走 `pt-snap skill`。
- helper 与泄漏/碎片/峰值诊断 skill 补充 focus 目标缺失但同目录或当前目录存在候选 `.db` 时的恢复规则:不按名称/大小/时间自动选库(即使只有一个候选也需用户确认),确认前不诊断、不用 Python/`sqlite3` 绕过 CLI,确认后先执行 `pt-snap metadata` 再继续原有预检;合法旧库的 `unavailable` / `metadata_missing` 不强制重新 import,换库不继承旧 device,且不持久化 focus、不代跑 import。
- `--template-info` 与 `get_template_info()` 展示参数 `choices`。`order_by` / `order_dir` 等会写入 SQL 标识符或关键字的参数必须声明封闭取值列表;字符串 `choices` 大小写不敏感,并规范化为声明拼写(如 `desc` → `DESC`)。
- `output_schema` 可声明字段语义(`units`、`metric_semantics`、`scope`、`denominator`、`sentinel`、`interpretation_limits`),查询级可声明 `semantics_version` 与解释限制。`--template-info` 与 `get_template_info()` 透出同一份契约。`execute_query()` 结果增加 `template` 与 `semantics_version`,行数据仍为原始 SQLite 值。优先覆盖 `leak_detection`、`memory_peak`、`allocator_gap`、`active_memory_callstack_at_event`。
Expand Down
20 changes: 18 additions & 2 deletions src/pt_snap_cli/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
from __future__ import annotations

import json
import logging
import shlex
import shutil
import sys
Expand Down Expand Up @@ -91,6 +92,9 @@
from pt_snap_cli.query.registry import discover_categories, get_query

AGENT_HELP_EPILOG = (
"Agents without pt-snap-helper can run "
"pt-snap skill install pt-snap-helper --json, then restart the agent "
"so the skill change takes effect. "
"Agents: prefer --json where supported. "
"Start with the pt-snap-helper skill; "
"use pt-snap capabilities --json and pt-snap overview --json before diagnosing; "
Expand Down Expand Up @@ -175,6 +179,17 @@ def _import_json(result: ImportResult) -> dict[str, JsonValue]:
)


def _import_snapshot_result(options: ImportOptions, *, json_output: bool) -> ImportResult:
if not json_output:
return ImportService().import_snapshot(options)
previous_disable = logging.root.manager.disable
logging.disable(logging.CRITICAL)

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Medium] 本 PR 唯一的产出代码行为变更缺少快速回归测试

logging.disable(CRITICAL) 用于抑制 snapshot runtime 写向 stderr 的进度日志(snapshot/util/logger.py:125 的 StreamHandler() 默认 stderr),但现有测试无法守住它:

  • _assert_clean_json_success(tests/test_cli_json.py:153-161)只校验 stdout,从不断言 result.stderr 为空;
  • test_import_json_fresh_and_reuse(tests/test_cli_json.py:397-420)继承同一缺口。

直接回退此处的抑制,pytest 仍然全绿,只有 3.13-only 的 clean-wheel 门禁才会失败(慢、需先构建 wheel)。建议补充针对 import --json(fresh + focused 路径)的空 stderr 断言,并校验 logging.root.manager.disable 在调用后恢复原值。

try:
return ImportService().import_snapshot(options)
finally:
logging.disable(previous_disable)


def _split_json(result: SplitResult) -> dict[str, JsonValue]:
return json_success(
output=str(result.output),
Expand Down Expand Up @@ -422,14 +437,15 @@ def import_snapshot(
) -> None:
"""Import a PyTorch memory snapshot into a SQLite database."""
try:
result = ImportService().import_snapshot(
result = _import_snapshot_result(
ImportOptions(
snapshot_file=snapshot_file,
output_dir=output_dir,
device=device,
set_focus=not no_focus,
force=force,
)
),
json_output=json_output,
)
except (
ImportToolMissingError,
Expand Down
21 changes: 17 additions & 4 deletions tests/skills/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,15 @@ enforced by `harness/descriptors.py` with unknown fields rejected.
declare `expect_output`, a mapping that must appear in the matched call's
recorded output before the action counts as matched. Actions may also declare
`status` (`success` or `error`; default `success`) so recovery cases can
require structured failures. Cases may cap the total call count with
require structured failures. Nested mappings and row lists in `expect_output`
are matched recursively. Cases may cap the total call count with
`expected_tools.max_calls`; zero enforces refusal-only cases.
- Descriptor paths are repository-relative and cannot contain `..`. Fixture
mount paths must be normalized absolute POSIX paths under `/fixtures/`.
mount paths must be unique, normalized absolute POSIX paths under `/fixtures/`.
`additional_fixtures` can mount more read-only synthetic SnapshotDBs when a
scenario depends on competing databases or devices. A semantic operation that
must publish an artifact may declare an explicit `writable_outputs` path under
`/outputs/`; it never makes project or fixture mounts writable.
- Diagnostic fixtures are declarative SQLite databases. Pickle inputs are never
materialized or exposed to a diagnostic runner.

Expand All @@ -40,7 +45,9 @@ python -m tests.skills validate tests/skills/suites/pt-snap-agent-e2e/suite.yaml

Agent adapters should expose only semantic operations through
`RecordingToolGateway`. The gateway records successful, failed, and denied
attempts and enforces the suite allowlist and call budget. Adapters remain
attempts and enforces the suite allowlist and call budget. Adapters should
raise `StructuredToolError` when a failed CLI call still has a JSON envelope;
the gateway retains that envelope in the graded trace. Adapters remain
responsible for process-level isolation: a fresh temporary working directory,
an isolated `HOME`, no network, a read-only project, read-only fixture mounts,
and cleared focus environment variables.
Expand Down Expand Up @@ -97,7 +104,10 @@ score.
scenarios plus sibling-database recovery when focus points at a missing file.
It is an `agent-cli` evaluation contract, not a shipped skill. Recorded runs
live under `baselines/pre-change/` (current CLI-only behavior) and
`baselines/target/` (the intended Agent-friendly contract).
`baselines/target/` (the post-change Agent-friendly contract). The comparison
requires higher task success, lower error-conclusion rate, and bounded mean
calls/output; recovery cases may use extra calls when a structured failure must
be corrected.

Comparison metrics are task success rate, mean call count, mean output bytes,
and error-conclusion rate:
Expand All @@ -111,6 +121,9 @@ python -m tests.skills baseline \

Diagnostic fixtures remain declarative SQLite databases. The import scenario
records `pt_snap.import` semantically and never materializes a pickle.
The release-gating clean-wheel check separately installs the built wheel into a
fresh venv, installs `pt-snap-helper` from package data, and completes import,
capability, overview, and peak-query discovery using JSON output.

Generated transcripts and reports belong under `.skill-evals/`, which is
ignored by Git. Normal `pytest` runs never invoke a live model.
3 changes: 2 additions & 1 deletion tests/skills/harness/__init__.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
"""Descriptor, fixture, runner, and grading primitives for skill evaluations."""

from .descriptors import DescriptorError, EvalCase, EvalSuite, SandboxPolicy, load_suite
from .gateway import RecordingToolGateway, ToolDeniedError
from .gateway import RecordingToolGateway, StructuredToolError, ToolDeniedError
from .grader import GradeResult, RunRecord, ToolCall, grade_run
from .metrics import BaselineSummary, CaseMetrics, collect_case_metrics, summarize_metrics

Expand All @@ -15,6 +15,7 @@
"RecordingToolGateway",
"RunRecord",
"SandboxPolicy",
"StructuredToolError",
"ToolCall",
"ToolDeniedError",
"collect_case_metrics",
Expand Down
98 changes: 74 additions & 24 deletions tests/skills/harness/descriptors.py
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,8 @@ class EvalCase:
required_evidence: tuple[tuple[str, str], ...]
mandatory_objectives: tuple[str, ...]
max_tool_calls: int | None = None
additional_fixtures: tuple[dict[str, Any], ...] = ()
writable_outputs: tuple[str, ...] = ()


@dataclass(frozen=True)
Expand Down Expand Up @@ -184,6 +186,38 @@ def _validate_dag(action_ids: set[str], edges: tuple[tuple[str, str], ...], cont
raise DescriptorError(f"{context} contains a cycle")


def _load_fixture(
raw_fixture: Any,
*,
path: Path,
suite_root: Path,
context: str,
) -> dict[str, Any]:
fixture = _strict_mapping(
raw_fixture,
context=context,
required={"builder", "definition", "mount_path", "read_only"},
allowed={"builder", "definition", "mount_path", "read_only"},
)
if fixture["builder"] != "synthetic-snapshotdb":
raise DescriptorError(f"{path}: only synthetic-snapshotdb fixtures are supported")
definition = _safe_path(suite_root, fixture["definition"], f"{context}.definition")
if not definition.is_file():
raise DescriptorError(f"{path}: fixture definition does not exist: {definition}")
if fixture["read_only"] is not True:
raise DescriptorError(f"{path}: diagnostic fixtures must be read-only")
mount_path = _string(fixture["mount_path"], f"{context}.mount_path")
if not mount_path.startswith("/"):
raise DescriptorError(f"{context}.mount_path must be an absolute POSIX path")
if ".." in PurePosixPath(mount_path).parts:
raise DescriptorError(f"{context}.mount_path cannot contain '..'")
if posixpath.normpath(mount_path) != mount_path:
raise DescriptorError(f"{context}.mount_path must be a normalized POSIX path")
if not mount_path.startswith("/fixtures/"):
raise DescriptorError(f"{context}.mount_path must be under /fixtures/")
return {**fixture, "definition_path": definition}


def _load_case(
path: Path,
*,
Expand Down Expand Up @@ -215,6 +249,8 @@ def _load_case(
"covers",
"task",
"fixture",
"additional_fixtures",
"writable_outputs",
"dialog",
"expected_tools",
"oracle",
Expand Down Expand Up @@ -383,33 +419,45 @@ def _load_case(
f"{path}: unknown objectives: {', '.join(sorted(unknown_objectives))}"
)

fixture: dict[str, Any] | None = None
if "fixture" in data:
raw_fixture = _strict_mapping(
fixture = (
_load_fixture(
data["fixture"],
path=path,
suite_root=suite_root,
context=f"{path}: fixture",
required={"builder", "definition", "mount_path", "read_only"},
allowed={"builder", "definition", "mount_path", "read_only"},
)
if raw_fixture["builder"] != "synthetic-snapshotdb":
raise DescriptorError(f"{path}: only synthetic-snapshotdb fixtures are supported")
definition = _safe_path(
suite_root, raw_fixture["definition"], f"{path}: fixture.definition"
)
if not definition.is_file():
raise DescriptorError(f"{path}: fixture definition does not exist: {definition}")
if raw_fixture["read_only"] is not True:
raise DescriptorError(f"{path}: diagnostic fixtures must be read-only")
mount_path = _string(raw_fixture["mount_path"], f"{path}: fixture.mount_path")
if not mount_path.startswith("/"):
raise DescriptorError(f"{path}: fixture.mount_path must be an absolute POSIX path")
if ".." in PurePosixPath(mount_path).parts:
raise DescriptorError(f"{path}: fixture.mount_path cannot contain '..'")
if posixpath.normpath(mount_path) != mount_path:
raise DescriptorError(f"{path}: fixture.mount_path must be a normalized POSIX path")
if not mount_path.startswith("/fixtures/"):
raise DescriptorError(f"{path}: fixture.mount_path must be under /fixtures/")
fixture = {**raw_fixture, "definition_path": definition}
if "fixture" in data
else None
)
additional_fixtures: list[dict[str, Any]] = []
if "additional_fixtures" in data:
raw_additional_fixtures = data["additional_fixtures"]
if not isinstance(raw_additional_fixtures, list):
raise DescriptorError(f"{path}: additional_fixtures must be a list")
for index, raw_fixture in enumerate(raw_additional_fixtures):
additional_fixtures.append(
_load_fixture(
raw_fixture,
path=path,
suite_root=suite_root,
context=f"{path}: additional_fixtures[{index}]",
)
)
mount_paths = [item["mount_path"] for item in additional_fixtures]
if fixture is not None:
mount_paths.append(fixture["mount_path"])
if len(mount_paths) != len(set(mount_paths)):
raise DescriptorError(f"{path}: fixture mount paths must be unique")
writable_outputs = _string_list(data.get("writable_outputs", []), f"{path}: writable_outputs")
for output_path in writable_outputs:
if not output_path.startswith("/"):
raise DescriptorError(f"{path}: writable output must be an absolute POSIX path")
if ".." in PurePosixPath(output_path).parts:
raise DescriptorError(f"{path}: writable output cannot contain '..'")
if posixpath.normpath(output_path) != output_path:
raise DescriptorError(f"{path}: writable output must be a normalized POSIX path")
if output_path != "/outputs" and not output_path.startswith("/outputs/"):
raise DescriptorError(f"{path}: writable output must be under /outputs/")

return EvalCase(
id=_string(data["id"], f"{path}: id"),
Expand All @@ -429,6 +477,8 @@ def _load_case(
required_evidence=tuple(required_evidence),
mandatory_objectives=mandatory_objectives,
max_tool_calls=max_tool_calls,
additional_fixtures=tuple(additional_fixtures),
writable_outputs=writable_outputs,
)


Expand Down
4 changes: 2 additions & 2 deletions tests/skills/harness/fixtures.py
Original file line number Diff line number Diff line change
Expand Up @@ -57,8 +57,8 @@ def load_fixture_definition(path: Path) -> dict[str, Any]:
raise DescriptorError(f"{path} has invalid fixture fields")
if data["schema_version"] != 1 or data["kind"] != "synthetic-snapshotdb":
raise DescriptorError(f"{path} must declare synthetic-snapshotdb schema version 1")
if not isinstance(data["devices"], dict) or not data["devices"]:
raise DescriptorError(f"{path}: devices must be a non-empty mapping")
if not isinstance(data["devices"], dict):

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Low] devices 非空校验对所有 fixture 放宽,而非仅限新增的空场景

移除 or not data["devices"] 后,任何 synthetic fixture 里意外写空的 devices: 都会静默构建出无设备的 SnapshotDB,而不是 fail fast。实际只有 tests/skills/suites/pt-snap-agent-e2e/fixtures/no-devices.yaml 需要该行为。建议:默认仍要求非空映射,为空设备场景增加显式 opt-in(如 fixture 级 allow_empty_devices 标志)。

raise DescriptorError(f"{path}: devices must be a mapping")
for device_id, device in data["devices"].items():
if not str(device_id).isdigit() or not isinstance(device, dict):
raise DescriptorError(f"{path}: device IDs must be non-negative integers")
Expand Down
18 changes: 18 additions & 0 deletions tests/skills/harness/gateway.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,12 @@ class ToolDeniedError(PermissionError):
"""Raised after a disallowed operation is recorded in the run trace."""


class StructuredToolError(RuntimeError):
def __init__(self, message: str, output: Any) -> None:
super().__init__(message)
self.output = output


ToolExecutor = Callable[[str, dict[str, Any]], Any]


Expand Down Expand Up @@ -47,6 +53,18 @@ def call(self, operation: str, arguments: dict[str, Any]) -> Any:

try:
output = self.executor(operation, dict(arguments))
except StructuredToolError as exc:
self._calls.append(
ToolCall(
call_id,
operation,
dict(arguments),
status="error",
output=exc.output,
error=f"{type(exc).__name__}: {exc}",
)
)
raise
except Exception as exc:
self._calls.append(
ToolCall(
Expand Down
23 changes: 20 additions & 3 deletions tests/skills/harness/grader.py
Original file line number Diff line number Diff line change
Expand Up @@ -88,13 +88,30 @@ def _is_subset(expected: dict[str, Any], actual: dict[str, Any]) -> bool:
return all(key in actual and actual[key] == value for key, value in expected.items())


def _value_contains(expected: Any, actual: Any) -> bool:
if isinstance(expected, dict):
return isinstance(actual, dict) and all(
key in actual and _value_contains(value, actual[key]) for key, value in expected.items()
)
if isinstance(expected, list):
return (
isinstance(actual, list)
and len(expected) <= len(actual)
and all(
_value_contains(item, candidate)
for item, candidate in zip(expected, actual, strict=False)

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Low] expect_output 列表匹配是按位置的前缀匹配,文档未说明

zip(expected, actual, strict=False) 要求期望行按索引与实际行对齐:行集合相同但顺序不同的查询结果会被判为不匹配。当前基线是确定性的,属潜在风险(基线若从 live CLI 输出重新生成且字段顺序变化会静默失败)。tests/skills/README.md:25-26 只说 "matched recursively",未提及顺序敏感。建议:在 README 注明位置匹配语义,或将行列表改为无序子集匹配(每个期望行在未使用的实际行中找到即可)。

)
)
return expected == actual


def _output_contains(expected: dict[str, Any], output: Any) -> bool:
if not expected:
return True
if isinstance(output, dict):
return _is_subset(expected, output)
if _value_contains(expected, output):
return True
if isinstance(output, list):
return any(_output_contains(expected, item) for item in output)
return any(_value_contains(expected, item) for item in output)
return False


Expand Down
5 changes: 5 additions & 0 deletions tests/skills/schemas/case-v1.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,11 @@ fields:
builder_values: [synthetic-snapshotdb]
mount_path_rule: normalized-absolute-posix-path-under-/fixtures/
read_only: true
additional_fixtures:
item_required: [builder, definition, mount_path, read_only]
rule: optional-distinct-read-only-synthetic-snapshotdb-mounts
writable_outputs:
rule: optional-distinct-normalized-absolute-paths-under-/outputs/
expected_tools:
required: [actions, partial_order, forbidden]
optional: [max_calls]
Expand Down
Loading
Loading