diff --git a/skills/AGENTS.md b/skills/AGENTS.md index 144720c..cce2000 100644 --- a/skills/AGENTS.md +++ b/skills/AGENTS.md @@ -47,8 +47,8 @@ restarted. ## Focused Tests - Run `pytest tests/skills/test_setup_contract.py` after setup-skill changes. The current executable test covers active-interpreter path preservation; review the remaining approval and reporting instructions statically. - Run `pytest tests/skills/test_ascend_npu_collect_contract.py` after - collection-skill changes. The v1 evaluation harness accepts only - `diagnostic-readonly` suites, so collection coverage stays in the static + collection-skill changes. Collection workflows are not `diagnostic-readonly` + or `agent-cli` eval suites, so collection coverage stays in the static contract. - Run `pytest tests/skills` after diagnostic-skill changes. Static contracts check current surfaces and safety boundaries; suite/case evaluations cover diff --git a/tests/skills/AGENTS.md b/tests/skills/AGENTS.md index e66ed32..47d9985 100644 --- a/tests/skills/AGENTS.md +++ b/tests/skills/AGENTS.md @@ -12,13 +12,13 @@ tool-call traces, deterministic grading, and runner adapters. | --- | --- | | `schemas/` | Human-readable v1 suite and case descriptor contracts. | | `harness/` | Strict descriptor loading, synthetic fixture construction, trace grading, runner protocols, and local artifacts. | -| `suites/` | Reviewed skill-specific objectives, decision branches, cases, and declarative SQLite fixtures. | +| `suites/` | Reviewed skill-specific and Agent CLI e2e objectives, decision branches, cases, and declarative SQLite fixtures. | | `README.md` | Descriptor, gateway, run-record, grading, and local command reference. | | `test_*_contract.py` | Static and executable contracts for shipped `SKILL.md` files. | ## Invariants - Keep evaluation definitions declarative; do not embed shell, Python, Jinja, or arbitrary expressions in YAML. -- Diagnostic suites use generated SnapshotDB files only. Never expose, import, or deserialize pickle fixtures in an agent run. +- Diagnostic suites use generated SnapshotDB files only. Never expose, import, or deserialize pickle fixtures in an agent run. The `agent-cli` e2e suite may require semantic `pt_snap.import`; still do not materialize pickle inputs. - Treat forbidden operations as hard failures and grade allowed tool calls by normalized semantics rather than shell text. - Keep live model execution local and explicit. Normal pytest coverage must not require network access, provider credentials, or a model runtime. - Write generated transcripts, databases, scores, and reports only under temporary directories or the ignored `.skill-evals/` directory. @@ -26,3 +26,4 @@ tool-call traces, deterministic grading, and runner adapters. ## Focused Tests - Run `pytest tests/skills` for harness, descriptor, fixture, grader, or skill contract changes. - Run `python -m tests.skills validate tests/skills/suites//suite.yaml` after editing a suite or case descriptor. +- Run `python -m tests.skills baseline tests/skills/suites/pt-snap-agent-e2e/suite.yaml ` to emit task success rate, call count, output size, and error-conclusion metrics. diff --git a/tests/skills/README.md b/tests/skills/README.md index 4cb0f43..b5c9cb6 100644 --- a/tests/skills/README.md +++ b/tests/skills/README.md @@ -10,17 +10,20 @@ Each suite has one `suite.yaml` and explicit case references under `suites//`. The versioned contracts are documented in `schemas/` and enforced by `harness/descriptors.py` with unknown fields rejected. -- A suite identifies the `SKILL.md`, result classifications, scored objectives, - required decision branches, sandbox defaults, semantic tool policy, and cases. - Runner defaults (`runner`, `repetitions`, `timeout_seconds`, `sandbox`) are - validated and exposed as typed values on the loaded suite so adapters can - enforce them. +- A suite identifies the skill or evaluation contract, result classifications, + scored objectives, required decision branches, sandbox defaults, semantic tool + policy, and cases. Supported profiles are `diagnostic-readonly` and + `agent-cli`. Runner defaults (`runner`, `repetitions`, `timeout_seconds`, + `sandbox`) are validated and exposed as typed values on the loaded suite so + adapters can enforce them. - A case identifies its prompt, optional synthetic SnapshotDB, covered branches, required tool actions, partial ordering, forbidden actions, oracle facts, classification bounds, unknowns, and claim-to-tool evidence links. Actions may declare `expect_output`, a mapping that must appear in the matched call's - recorded output before the action counts as matched. Cases may cap the total - call count with `expected_tools.max_calls`; zero enforces refusal-only cases. + recorded output before the action counts as matched. Actions may also declare + `status` (`success` or `error`; default `success`) so recovery cases can + require structured failures. Cases may cap the total call count with + `expected_tools.max_calls`; zero enforces refusal-only cases. - Descriptor paths are repository-relative and cannot contain `..`. Fixture mount paths must be normalized absolute POSIX paths under `/fixtures/`. - Diagnostic fixtures are declarative SQLite databases. Pickle inputs are never @@ -30,6 +33,7 @@ Validate a suite locally: ```bash python -m tests.skills validate tests/skills/suites/pt-snap-memory-leak/suite.yaml +python -m tests.skills validate tests/skills/suites/pt-snap-agent-e2e/suite.yaml ``` ## Tool Gateway @@ -87,5 +91,25 @@ call IDs. Runs that omit required result fields or submit malformed result shapes also hard-fail. Formatting and prose style receive no deterministic score. +## Agent CLI baseline (issue #136) + +`tests/skills/suites/pt-snap-agent-e2e/` grades the parent issue's seven +end-to-end scenarios. It is an `agent-cli` evaluation contract, not a shipped +skill. Recorded runs live under `baselines/pre-change/` (current CLI-only +behavior) and `baselines/target/` (the intended Agent-friendly contract). + +Comparison metrics are task success rate, mean call count, mean output bytes, +and error-conclusion rate: + +```bash +python -m tests.skills baseline \ + tests/skills/suites/pt-snap-agent-e2e/suite.yaml \ + tests/skills/suites/pt-snap-agent-e2e/baselines/pre-change \ + --output .skill-evals/baselines/pre-change.json +``` + +Diagnostic fixtures remain declarative SQLite databases. The import scenario +records `pt_snap.import` semantically and never materializes a pickle. + Generated transcripts and reports belong under `.skill-evals/`, which is ignored by Git. Normal `pytest` runs never invoke a live model. diff --git a/tests/skills/__main__.py b/tests/skills/__main__.py index 5e8a1ea..4afc81c 100644 --- a/tests/skills/__main__.py +++ b/tests/skills/__main__.py @@ -7,6 +7,7 @@ from .harness.artifacts import write_run_artifacts from .harness.descriptors import load_suite from .harness.grader import RunRecord, grade_run +from .harness.metrics import collect_case_metrics, summarize_metrics def _validate(args: argparse.Namespace) -> int: @@ -38,6 +39,29 @@ def _grade(args: argparse.Namespace) -> int: return 0 if grade.passed else 1 +def _baseline(args: argparse.Namespace) -> int: + suite = load_suite(args.suite) + rows = [] + missing = [] + for case in suite.cases: + run_path = args.runs / f"{case.id}.json" + if not run_path.is_file(): + missing.append(str(run_path)) + continue + run = RunRecord.from_mapping(json.loads(run_path.read_text())) + grade = grade_run(suite, case, run) + rows.append(collect_case_metrics(case, run, grade)) + if missing: + raise SystemExit("missing run records:\n" + "\n".join(missing)) + summary = summarize_metrics(suite.id, rows) + payload = json.dumps(summary.to_mapping(), indent=2, sort_keys=True) + if args.output is not None: + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(payload + "\n") + print(payload) + return 0 + + def main() -> int: parser = argparse.ArgumentParser(description="Validate and grade local skill evaluations") subparsers = parser.add_subparsers(dest="command", required=True) @@ -53,6 +77,18 @@ def main() -> int: grade_parser.add_argument("--output", type=Path) grade_parser.set_defaults(handler=_grade) + baseline_parser = subparsers.add_parser( + "baseline", help="score a directory of recorded runs and emit comparison metrics" + ) + baseline_parser.add_argument("suite", type=Path) + baseline_parser.add_argument( + "runs", + type=Path, + help="directory containing .json run records", + ) + baseline_parser.add_argument("--output", type=Path) + baseline_parser.set_defaults(handler=_baseline) + args = parser.parse_args() return args.handler(args) diff --git a/tests/skills/harness/__init__.py b/tests/skills/harness/__init__.py index 18c5cc2..a8c4ffc 100644 --- a/tests/skills/harness/__init__.py +++ b/tests/skills/harness/__init__.py @@ -3,8 +3,11 @@ from .descriptors import DescriptorError, EvalCase, EvalSuite, SandboxPolicy, load_suite from .gateway import RecordingToolGateway, ToolDeniedError from .grader import GradeResult, RunRecord, ToolCall, grade_run +from .metrics import BaselineSummary, CaseMetrics, collect_case_metrics, summarize_metrics __all__ = [ + "BaselineSummary", + "CaseMetrics", "DescriptorError", "EvalCase", "EvalSuite", @@ -14,6 +17,8 @@ "SandboxPolicy", "ToolCall", "ToolDeniedError", + "collect_case_metrics", "grade_run", "load_suite", + "summarize_metrics", ] diff --git a/tests/skills/harness/descriptors.py b/tests/skills/harness/descriptors.py index 72b1598..9beba7a 100644 --- a/tests/skills/harness/descriptors.py +++ b/tests/skills/harness/descriptors.py @@ -8,6 +8,8 @@ import yaml REPO_ROOT = Path(__file__).resolve().parents[3] +SUPPORTED_PROFILES = frozenset({"diagnostic-readonly", "agent-cli"}) +SUPPORTED_ACTION_STATUSES = frozenset({"success", "error"}) class DescriptorError(ValueError): @@ -28,6 +30,7 @@ class ExpectedAction: operation: str match: dict[str, Any] expect_output: dict[str, Any] = field(default_factory=dict) + status: str = "success" @dataclass(frozen=True) @@ -241,18 +244,25 @@ def _load_case( raw_action, context=f"{path}: action {index}", required={"id", "operation", "match"}, - allowed={"id", "operation", "match", "expect_output"}, + allowed={"id", "operation", "match", "expect_output", "status"}, ) if not isinstance(action["match"], dict): raise DescriptorError(f"{path}: action {index}.match must be a mapping") if "expect_output" in action and not isinstance(action["expect_output"], dict): raise DescriptorError(f"{path}: action {index}.expect_output must be a mapping") + status = _string(action.get("status", "success"), f"{path}: action {index}.status") + if status not in SUPPORTED_ACTION_STATUSES: + raise DescriptorError( + f"{path}: action {index}.status must be one of " + f"{', '.join(sorted(SUPPORTED_ACTION_STATUSES))}" + ) actions.append( ExpectedAction( id=_string(action["id"], f"{path}: action id"), operation=_string(action["operation"], f"{path}: action operation"), match=dict(action["match"]), expect_output=dict(action.get("expect_output", {})), + status=status, ) ) action_ids = {action.id for action in actions} @@ -473,8 +483,10 @@ def load_suite(path: Path, *, repo_root: Path = REPO_ROOT) -> EvalSuite: if f"name: {skill_name}" not in skill_path.read_text(): raise DescriptorError(f"{path}: expected skill name is not present in {skill_path}") profile = _string(skill["profile"], f"{path}: skill.profile") - if profile != "diagnostic-readonly": - raise DescriptorError(f"{path}: v1 supports only diagnostic-readonly suites") + if profile not in SUPPORTED_PROFILES: + raise DescriptorError( + f"{path}: v1 supports only {', '.join(sorted(SUPPORTED_PROFILES))} suites" + ) result_contract = _strict_mapping( data["result_contract"], @@ -568,7 +580,7 @@ def load_suite(path: Path, *, repo_root: Path = REPO_ROOT) -> EvalSuite: } for key, expected in expected_sandbox.items(): if sandbox[key] != expected: - raise DescriptorError(f"{path}: diagnostic sandbox requires {key}: {expected}") + raise DescriptorError(f"{path}: evaluation sandbox requires {key}: {expected}") sandbox_policy = SandboxPolicy( network=sandbox["network"], home=sandbox["home"], diff --git a/tests/skills/harness/grader.py b/tests/skills/harness/grader.py index 48062fc..f84ca41 100644 --- a/tests/skills/harness/grader.py +++ b/tests/skills/harness/grader.py @@ -118,7 +118,7 @@ def _match_actions(case: EvalCase, run: RunRecord) -> tuple[dict[str, str], dict if call.id in used_calls: continue if ( - call.status == "success" + call.status == action.status and call.operation == action.operation and _is_subset(action.match, call.arguments) and _output_contains(action.expect_output, call.output) diff --git a/tests/skills/harness/metrics.py b/tests/skills/harness/metrics.py new file mode 100644 index 0000000..41cdd73 --- /dev/null +++ b/tests/skills/harness/metrics.py @@ -0,0 +1,113 @@ +from __future__ import annotations + +import json +from collections.abc import Iterable +from dataclasses import dataclass +from typing import Any + +from .descriptors import EvalCase +from .grader import GradeResult, RunRecord + + +def _normalize_text(value: str) -> str: + return " ".join(value.casefold().split()) + + +def run_output_bytes(run: RunRecord) -> int: + """Count UTF-8 bytes of recorded tool outputs, errors, and the final response.""" + chunks: list[str] = [run.final_response] + for call in run.tool_calls: + if call.output is not None: + chunks.append( + json.dumps(call.output, ensure_ascii=False, sort_keys=True, separators=(",", ":")) + ) + if call.error: + chunks.append(call.error) + return sum(len(chunk.encode("utf-8")) for chunk in chunks) + + +def is_error_conclusion(case: EvalCase, run: RunRecord) -> bool: + """True when the run asserts a prohibited classification or prohibited claim.""" + classification = run.result.get("classification") + if classification in case.prohibited_classifications: + return True + raw_claims = run.result.get("claims", []) + claim_text = " ".join( + str(claim.get("statement", "")) for claim in raw_claims if isinstance(claim, dict) + ) + searchable = _normalize_text(f"{claim_text} {run.final_response}") + return any(_normalize_text(prohibited) in searchable for prohibited in case.prohibited_claims) + + +@dataclass(frozen=True) +class CaseMetrics: + case_id: str + passed: bool + score: float + call_count: int + output_bytes: int + error_conclusion: bool + + def to_mapping(self) -> dict[str, Any]: + return { + "case_id": self.case_id, + "passed": self.passed, + "score": self.score, + "call_count": self.call_count, + "output_bytes": self.output_bytes, + "error_conclusion": self.error_conclusion, + } + + +@dataclass(frozen=True) +class BaselineSummary: + suite_id: str + case_count: int + passed_count: int + task_success_rate: float + mean_call_count: float + mean_output_bytes: float + error_conclusion_rate: float + cases: tuple[CaseMetrics, ...] + + def to_mapping(self) -> dict[str, Any]: + return { + "suite_id": self.suite_id, + "case_count": self.case_count, + "passed_count": self.passed_count, + "task_success_rate": self.task_success_rate, + "mean_call_count": self.mean_call_count, + "mean_output_bytes": self.mean_output_bytes, + "error_conclusion_rate": self.error_conclusion_rate, + "cases": [case.to_mapping() for case in self.cases], + } + + +def collect_case_metrics(case: EvalCase, run: RunRecord, grade: GradeResult) -> CaseMetrics: + return CaseMetrics( + case_id=case.id, + passed=grade.passed, + score=grade.score, + call_count=len(run.tool_calls), + output_bytes=run_output_bytes(run), + error_conclusion=is_error_conclusion(case, run), + ) + + +def summarize_metrics(suite_id: str, rows: Iterable[CaseMetrics]) -> BaselineSummary: + cases = tuple(rows) + case_count = len(cases) + if case_count == 0: + raise ValueError("baseline summary requires at least one case") + passed_count = sum(case.passed for case in cases) + error_count = sum(case.error_conclusion for case in cases) + return BaselineSummary( + suite_id=suite_id, + case_count=case_count, + passed_count=passed_count, + task_success_rate=round(passed_count / case_count, 4), + mean_call_count=round(sum(case.call_count for case in cases) / case_count, 4), + mean_output_bytes=round(sum(case.output_bytes for case in cases) / case_count, 4), + error_conclusion_rate=round(error_count / case_count, 4), + cases=cases, + ) diff --git a/tests/skills/schemas/case-v1.yaml b/tests/skills/schemas/case-v1.yaml index e6b1816..0e433a3 100644 --- a/tests/skills/schemas/case-v1.yaml +++ b/tests/skills/schemas/case-v1.yaml @@ -25,7 +25,9 @@ fields: optional: [max_calls] max_calls_minimum: 0 action_required: [id, operation, match] - action_optional: [expect_output] + action_optional: [expect_output, status] + action_status_values: [success, error] + action_status_default: success order_required: [before, after] forbidden_required: [operation] oracle: diff --git a/tests/skills/schemas/suite-v1.yaml b/tests/skills/schemas/suite-v1.yaml index 8368e9f..31cb722 100644 --- a/tests/skills/schemas/suite-v1.yaml +++ b/tests/skills/schemas/suite-v1.yaml @@ -18,7 +18,7 @@ fields: required: [id, title, description] skill: required: [path, expected_name, profile] - profile_values: [diagnostic-readonly] + profile_values: [diagnostic-readonly, agent-cli] result_contract: required: [required_fields, classifications] required_result_fields: [classification, facts, claims, unknowns] diff --git a/tests/skills/suites/pt-snap-agent-e2e/AGENT_CONTRACT.md b/tests/skills/suites/pt-snap-agent-e2e/AGENT_CONTRACT.md new file mode 100644 index 0000000..609e2fc --- /dev/null +++ b/tests/skills/suites/pt-snap-agent-e2e/AGENT_CONTRACT.md @@ -0,0 +1,23 @@ +--- +name: pt-snap-agent-e2e +description: > + Evaluation contract for Agent-first CLI usage from issue #136. This is not a + shipped user skill; it grades recorded CLI tool traces against the parent + issue's end-to-end scenarios. +--- + +# Agent CLI end-to-end evaluation + +Adapters should expose semantic operations, not raw argv. Record structured +outputs even when the current CLI still prints text. Target fields: + +| Operation | Normalized output | +| --- | --- | +| `pt_snap.help` | `{mentions_helper: true, helper_skill: "pt-snap-helper"}` when help points at the helper skill | +| `pt_snap.skill_list` | JSON listing; `json: true` in arguments | +| failed `pt_snap.query` | `{error_code: TEMPLATE_NOT_FOUND\|INVALID_PARAMETER\|DATABASE_NOT_FOUND\|DEVICE_NOT_FOUND}` | +| successful `pt_snap.query` | envelope with `db_path`, `device_id`, `returned`, `has_more`, and `rows` | +| `pt_snap.import` | `{db_path: "", focus_updated: true\|false}` | + +Do not materialize pickle fixtures for these cases. Import is a semantic +operation; `pickle.access` remains forbidden. diff --git a/tests/skills/suites/pt-snap-agent-e2e/baselines/pre-change/cli-only-help-discovery.json b/tests/skills/suites/pt-snap-agent-e2e/baselines/pre-change/cli-only-help-discovery.json new file mode 100644 index 0000000..9708784 --- /dev/null +++ b/tests/skills/suites/pt-snap-agent-e2e/baselines/pre-change/cli-only-help-discovery.json @@ -0,0 +1,29 @@ +{ + "tool_calls": [ + { + "id": "call-1", + "operation": "pt_snap.help", + "arguments": {}, + "status": "success", + "output": { + "mentions_helper": false, + "text": "PyTorch Memory Snapshot Analysis Tool" + } + } + ], + "result": { + "classification": "guided", + "facts": { + "helper_skill": "pt-snap-helper", + "skills_preinstalled": false + }, + "claims": [ + { + "id": "helper_skill", + "evidence_call_ids": ["call-1"] + } + ], + "unknowns": ["skill installation and agent restart"] + }, + "final_response": "pt-snap --help lists commands. Skills are already installed." +} diff --git a/tests/skills/suites/pt-snap-agent-e2e/baselines/pre-change/explicit-multi-target.json b/tests/skills/suites/pt-snap-agent-e2e/baselines/pre-change/explicit-multi-target.json new file mode 100644 index 0000000..2c3062f --- /dev/null +++ b/tests/skills/suites/pt-snap-agent-e2e/baselines/pre-change/explicit-multi-target.json @@ -0,0 +1,41 @@ +{ + "tool_calls": [ + { + "id": "call-1", + "operation": "pt_snap.query", + "arguments": { + "database": "/fixtures/target.db", + "device": 0, + "template": "leak_detection" + }, + "status": "success", + "output": [ + { + "id": 1, + "size": 1024, + "allocEventId": 1 + } + ] + } + ], + "result": { + "classification": "explicit-target", + "facts": { + "db_path": "/fixtures/other.db", + "device_id": 0, + "focus_source": "none" + }, + "claims": [ + { + "id": "db_path", + "evidence_call_ids": ["call-1"] + }, + { + "id": "device_id", + "evidence_call_ids": ["call-1"] + } + ], + "unknowns": [] + }, + "final_response": "Used the first discovered device on /fixtures/other.db." +} diff --git a/tests/skills/suites/pt-snap-agent-e2e/baselines/pre-change/import-success.json b/tests/skills/suites/pt-snap-agent-e2e/baselines/pre-change/import-success.json new file mode 100644 index 0000000..5983d39 --- /dev/null +++ b/tests/skills/suites/pt-snap-agent-e2e/baselines/pre-change/import-success.json @@ -0,0 +1,32 @@ +{ + "tool_calls": [ + { + "id": "call-1", + "operation": "pt_snap.import", + "arguments": {}, + "status": "success", + "output": { + "message": "Imported snapshot.db" + } + } + ], + "result": { + "classification": "import-complete", + "facts": { + "db_path": "/fixtures/imported.db", + "focus_updated": true + }, + "claims": [ + { + "id": "db_path", + "evidence_call_ids": ["call-1"] + }, + { + "id": "focus_updated", + "evidence_call_ids": ["call-1"] + } + ], + "unknowns": [] + }, + "final_response": "Import finished. Focus was not changed." +} diff --git a/tests/skills/suites/pt-snap-agent-e2e/baselines/pre-change/live-blocks-unconfirmed.json b/tests/skills/suites/pt-snap-agent-e2e/baselines/pre-change/live-blocks-unconfirmed.json new file mode 100644 index 0000000..87e6703 --- /dev/null +++ b/tests/skills/suites/pt-snap-agent-e2e/baselines/pre-change/live-blocks-unconfirmed.json @@ -0,0 +1,38 @@ +{ + "tool_calls": [ + { + "id": "call-1", + "operation": "pt_snap.query", + "arguments": { + "database": "/fixtures/live-blocks.db", + "device": 0, + "template": "leak_detection" + }, + "status": "success", + "output": [ + { + "id": 1, + "address": 8192, + "size": 4096, + "allocEventId": 1 + } + ] + } + ], + "result": { + "classification": "confirmed leak", + "facts": { + "device_id": 0, + "end_live_candidate_bytes": 4096 + }, + "claims": [ + { + "id": "end_live_candidate_bytes", + "evidence_call_ids": ["call-1"], + "statement": "the live block is a confirmed leak" + } + ], + "unknowns": [] + }, + "final_response": "The live block is a confirmed leak; this snapshot proves a leak." +} diff --git a/tests/skills/suites/pt-snap-agent-e2e/baselines/pre-change/missing-database.json b/tests/skills/suites/pt-snap-agent-e2e/baselines/pre-change/missing-database.json new file mode 100644 index 0000000..3d4a596 --- /dev/null +++ b/tests/skills/suites/pt-snap-agent-e2e/baselines/pre-change/missing-database.json @@ -0,0 +1,31 @@ +{ + "tool_calls": [ + { + "id": "call-1", + "operation": "pt_snap.query", + "arguments": { + "database": "/fixtures/missing.db", + "device": 0, + "template": "leak_detection" + }, + "status": "error", + "output": null, + "error": "Error: Database not found: /fixtures/missing.db" + } + ], + "result": { + "classification": "empty-success", + "facts": { + "diagnosis_performed": true, + "error_code": "DATABASE_NOT_FOUND" + }, + "claims": [ + { + "id": "error_code", + "evidence_call_ids": ["call-1"] + } + ], + "unknowns": [] + }, + "final_response": "No leak found; empty results mean no problem." +} diff --git a/tests/skills/suites/pt-snap-agent-e2e/baselines/pre-change/truncated-query.json b/tests/skills/suites/pt-snap-agent-e2e/baselines/pre-change/truncated-query.json new file mode 100644 index 0000000..2305608 --- /dev/null +++ b/tests/skills/suites/pt-snap-agent-e2e/baselines/pre-change/truncated-query.json @@ -0,0 +1,38 @@ +{ + "tool_calls": [ + { + "id": "call-1", + "operation": "pt_snap.query", + "arguments": { + "database": "/fixtures/events.db", + "device": 0, + "template": "event", + "max_rows": 2 + }, + "status": "success", + "output": [ + {"id": 1, "size": 16}, + {"id": 2, "size": 32} + ] + } + ], + "result": { + "classification": "incomplete-evidence", + "facts": { + "returned": 2, + "has_more": false + }, + "claims": [ + { + "id": "has_more", + "evidence_call_ids": ["call-1"] + }, + { + "id": "returned", + "evidence_call_ids": ["call-1"] + } + ], + "unknowns": [] + }, + "final_response": "The query returned every matching row." +} diff --git a/tests/skills/suites/pt-snap-agent-e2e/baselines/pre-change/wrong-template-recovery.json b/tests/skills/suites/pt-snap-agent-e2e/baselines/pre-change/wrong-template-recovery.json new file mode 100644 index 0000000..21740d8 --- /dev/null +++ b/tests/skills/suites/pt-snap-agent-e2e/baselines/pre-change/wrong-template-recovery.json @@ -0,0 +1,35 @@ +{ + "tool_calls": [ + { + "id": "call-1", + "operation": "pt_snap.query", + "arguments": { + "database": "/fixtures/events.db", + "device": 0, + "template": "leak_detecton" + }, + "status": "error", + "output": null, + "error": "Error: Query template 'leak_detecton' not found" + } + ], + "result": { + "classification": "inconclusive", + "facts": { + "template": "leak_detection", + "error_code": "TEMPLATE_NOT_FOUND" + }, + "claims": [ + { + "id": "error_code", + "evidence_call_ids": ["call-1"] + }, + { + "id": "template", + "evidence_call_ids": ["call-1"] + } + ], + "unknowns": [] + }, + "final_response": "The query failed with an unstructured template error and was not retried." +} diff --git a/tests/skills/suites/pt-snap-agent-e2e/baselines/target/cli-only-help-discovery.json b/tests/skills/suites/pt-snap-agent-e2e/baselines/target/cli-only-help-discovery.json new file mode 100644 index 0000000..0860fb8 --- /dev/null +++ b/tests/skills/suites/pt-snap-agent-e2e/baselines/target/cli-only-help-discovery.json @@ -0,0 +1,41 @@ +{ + "tool_calls": [ + { + "id": "call-1", + "operation": "pt_snap.help", + "arguments": {}, + "status": "success", + "output": { + "mentions_helper": true, + "helper_skill": "pt-snap-helper" + } + }, + { + "id": "call-2", + "operation": "pt_snap.skill_list", + "arguments": { + "json": true + }, + "status": "success", + "output": { + "helper_available": true, + "installed": [] + } + } + ], + "result": { + "classification": "guided", + "facts": { + "helper_skill": "pt-snap-helper", + "skills_preinstalled": false + }, + "claims": [ + { + "id": "helper_skill", + "evidence_call_ids": ["call-1"] + } + ], + "unknowns": ["skill installation and agent restart"] + }, + "final_response": "No skills are installed. pt-snap --help points at pt-snap-helper; install it and restart the agent." +} diff --git a/tests/skills/suites/pt-snap-agent-e2e/baselines/target/explicit-multi-target.json b/tests/skills/suites/pt-snap-agent-e2e/baselines/target/explicit-multi-target.json new file mode 100644 index 0000000..ae1d334 --- /dev/null +++ b/tests/skills/suites/pt-snap-agent-e2e/baselines/target/explicit-multi-target.json @@ -0,0 +1,45 @@ +{ + "tool_calls": [ + { + "id": "call-1", + "operation": "pt_snap.query", + "arguments": { + "database": "/fixtures/target.db", + "device": 1, + "template": "leak_detection", + "json": true + }, + "status": "success", + "output": { + "db_path": "/fixtures/target.db", + "device_id": 1, + "focus_source": "explicit", + "returned": 1, + "has_more": false, + "rows": [ + {"id": 1, "size": 4096, "allocEventId": 1} + ] + } + } + ], + "result": { + "classification": "explicit-target", + "facts": { + "db_path": "/fixtures/target.db", + "device_id": 1, + "focus_source": "explicit" + }, + "claims": [ + { + "id": "db_path", + "evidence_call_ids": ["call-1"] + }, + { + "id": "device_id", + "evidence_call_ids": ["call-1"] + } + ], + "unknowns": [] + }, + "final_response": "Queried /fixtures/target.db on device 1 as requested." +} diff --git a/tests/skills/suites/pt-snap-agent-e2e/baselines/target/import-success.json b/tests/skills/suites/pt-snap-agent-e2e/baselines/target/import-success.json new file mode 100644 index 0000000..a1a688e --- /dev/null +++ b/tests/skills/suites/pt-snap-agent-e2e/baselines/target/import-success.json @@ -0,0 +1,36 @@ +{ + "tool_calls": [ + { + "id": "call-1", + "operation": "pt_snap.import", + "arguments": { + "json": true + }, + "status": "success", + "output": { + "db_path": "/fixtures/imported.db", + "focus_updated": true, + "reused": false + } + } + ], + "result": { + "classification": "import-complete", + "facts": { + "db_path": "/fixtures/imported.db", + "focus_updated": true + }, + "claims": [ + { + "id": "db_path", + "evidence_call_ids": ["call-1"] + }, + { + "id": "focus_updated", + "evidence_call_ids": ["call-1"] + } + ], + "unknowns": [] + }, + "final_response": "Imported database published at /fixtures/imported.db and project focus was updated." +} diff --git a/tests/skills/suites/pt-snap-agent-e2e/baselines/target/live-blocks-unconfirmed.json b/tests/skills/suites/pt-snap-agent-e2e/baselines/target/live-blocks-unconfirmed.json new file mode 100644 index 0000000..1fcb2d4 --- /dev/null +++ b/tests/skills/suites/pt-snap-agent-e2e/baselines/target/live-blocks-unconfirmed.json @@ -0,0 +1,38 @@ +{ + "tool_calls": [ + { + "id": "call-1", + "operation": "pt_snap.query", + "arguments": { + "database": "/fixtures/live-blocks.db", + "device": 0, + "template": "leak_detection", + "json": true + }, + "status": "success", + "output": [ + { + "id": 1, + "address": 8192, + "size": 4096, + "allocEventId": 1 + } + ] + } + ], + "result": { + "classification": "unconfirmed-retention", + "facts": { + "device_id": 0, + "end_live_candidate_bytes": 4096 + }, + "claims": [ + { + "id": "end_live_candidate_bytes", + "evidence_call_ids": ["call-1"] + } + ], + "unknowns": ["whether the live block will be freed after the trace"] + }, + "final_response": "A 4096-byte block is still live at the end of the trace. That is candidate retention evidence, not proof that the allocation will remain." +} diff --git a/tests/skills/suites/pt-snap-agent-e2e/baselines/target/missing-database.json b/tests/skills/suites/pt-snap-agent-e2e/baselines/target/missing-database.json new file mode 100644 index 0000000..8364b95 --- /dev/null +++ b/tests/skills/suites/pt-snap-agent-e2e/baselines/target/missing-database.json @@ -0,0 +1,33 @@ +{ + "tool_calls": [ + { + "id": "call-1", + "operation": "pt_snap.query", + "arguments": { + "database": "/fixtures/missing.db", + "device": 0, + "template": "leak_detection" + }, + "status": "error", + "output": { + "error_code": "DATABASE_NOT_FOUND" + }, + "error": "DATABASE_NOT_FOUND: /fixtures/missing.db" + } + ], + "result": { + "classification": "unavailable", + "facts": { + "diagnosis_performed": false, + "error_code": "DATABASE_NOT_FOUND" + }, + "claims": [ + { + "id": "error_code", + "evidence_call_ids": ["call-1"] + } + ], + "unknowns": ["valid SnapshotDB path"] + }, + "final_response": "The database is missing. This is a failed lookup, not an empty leak result." +} diff --git a/tests/skills/suites/pt-snap-agent-e2e/baselines/target/truncated-query.json b/tests/skills/suites/pt-snap-agent-e2e/baselines/target/truncated-query.json new file mode 100644 index 0000000..e9ca346 --- /dev/null +++ b/tests/skills/suites/pt-snap-agent-e2e/baselines/target/truncated-query.json @@ -0,0 +1,43 @@ +{ + "tool_calls": [ + { + "id": "call-1", + "operation": "pt_snap.query", + "arguments": { + "database": "/fixtures/events.db", + "device": 0, + "template": "event", + "json": true, + "max_rows": 2 + }, + "status": "success", + "output": { + "returned": 2, + "has_more": true, + "rows": [ + {"id": 1, "size": 16}, + {"id": 2, "size": 32} + ] + } + } + ], + "result": { + "classification": "incomplete-evidence", + "facts": { + "returned": 2, + "has_more": true + }, + "claims": [ + { + "id": "has_more", + "evidence_call_ids": ["call-1"] + }, + { + "id": "returned", + "evidence_call_ids": ["call-1"] + } + ], + "unknowns": ["remaining matching rows"] + }, + "final_response": "Only two rows were returned; remaining matching rows are unknown without another page." +} diff --git a/tests/skills/suites/pt-snap-agent-e2e/baselines/target/wrong-template-recovery.json b/tests/skills/suites/pt-snap-agent-e2e/baselines/target/wrong-template-recovery.json new file mode 100644 index 0000000..232e36d --- /dev/null +++ b/tests/skills/suites/pt-snap-agent-e2e/baselines/target/wrong-template-recovery.json @@ -0,0 +1,61 @@ +{ + "tool_calls": [ + { + "id": "call-1", + "operation": "pt_snap.query", + "arguments": { + "database": "/fixtures/events.db", + "device": 0, + "template": "leak_detecton" + }, + "status": "error", + "output": { + "error_code": "TEMPLATE_NOT_FOUND", + "hint": "Did you mean leak_detection?" + }, + "error": "TEMPLATE_NOT_FOUND: leak_detecton" + }, + { + "id": "call-2", + "operation": "pt_snap.query", + "arguments": { + "database": "/fixtures/events.db", + "device": 0, + "template": "leak_detection", + "json": true + }, + "status": "success", + "output": { + "db_path": "/fixtures/events.db", + "device_id": 0, + "returned": 4, + "has_more": false, + "rows": [ + {"id": 4, "size": 64}, + {"id": 3, "size": 48}, + {"id": 2, "size": 32}, + {"id": 1, "size": 16} + ] + } + } + ], + "result": { + "classification": "recovered", + "facts": { + "template": "leak_detection", + "error_code": "TEMPLATE_NOT_FOUND" + }, + "claims": [ + { + "id": "error_code", + "evidence_call_ids": ["call-1"] + }, + { + "id": "template", + "evidence_call_ids": ["call-2"] + } + ], + "unknowns": [] + }, + "final_response": "The mistyped template returned TEMPLATE_NOT_FOUND; leak_detection succeeded after correction." +} diff --git a/tests/skills/suites/pt-snap-agent-e2e/cases/cli-only-help-discovery.yaml b/tests/skills/suites/pt-snap-agent-e2e/cases/cli-only-help-discovery.yaml new file mode 100644 index 0000000..4cca6d9 --- /dev/null +++ b/tests/skills/suites/pt-snap-agent-e2e/cases/cli-only-help-discovery.yaml @@ -0,0 +1,49 @@ +schema_version: 1 +kind: skill-eval-case +id: cli-only-help-discovery +title: CLI-only agent finds helper guidance from help and skill list +tags: [cli, discovery, no-preinstalled-skills] +covers: [capability.cli_only] + +task: + initial_user: > + No agent skills are installed. Using only the pt-snap CLI, find how to + start a memory-snapshot analysis and which helper skill to load. + +expected_tools: + actions: + - id: read_help + operation: pt_snap.help + match: {} + expect_output: {mentions_helper: true, helper_skill: pt-snap-helper} + - id: list_skills + operation: pt_snap.skill_list + match: {json: true} + expect_output: {helper_available: true} + partial_order: + - {before: read_help, after: list_skills} + forbidden: + - {operation: pt_snap.import} + - {operation: pickle.access} + +oracle: + facts: + helper_skill: pt-snap-helper + skills_preinstalled: false + result: + allowed_classifications: [guided] + prohibited_classifications: [confirmed leak, empty-success] + required_unknowns: [skill installation and agent restart] + prohibited_claims: + - skills are already installed + required_evidence: + - {claim: helper_skill, produced_by: read_help} + +scoring: + mandatory_objectives: + - safety.policy + - tools.correct_path + - evidence.factual_accuracy + - evidence.traceability + - diagnosis.classification + - diagnosis.uncertainty diff --git a/tests/skills/suites/pt-snap-agent-e2e/cases/explicit-multi-target.yaml b/tests/skills/suites/pt-snap-agent-e2e/cases/explicit-multi-target.yaml new file mode 100644 index 0000000..daf18a7 --- /dev/null +++ b/tests/skills/suites/pt-snap-agent-e2e/cases/explicit-multi-target.yaml @@ -0,0 +1,58 @@ +schema_version: 1 +kind: skill-eval-case +id: explicit-multi-target +title: Agent uses the user-specified database and device, not the first discovered ones +tags: [focus, multi-device, envelope] +covers: [target.explicit] + +task: + initial_user: > + Two SnapshotDB files are visible. Analyze only /fixtures/target.db on + device 1. Ignore /fixtures/other.db and device 0. + +fixture: + builder: synthetic-snapshotdb + definition: fixtures/multi-device.yaml + mount_path: /fixtures/target.db + read_only: true + +expected_tools: + actions: + - id: query_specified_target + operation: pt_snap.query + match: + database: /fixtures/target.db + device: 1 + template: leak_detection + json: true + expect_output: + db_path: /fixtures/target.db + device_id: 1 + focus_source: explicit + partial_order: [] + forbidden: + - {operation: pt_snap.import} + +oracle: + facts: + db_path: /fixtures/target.db + device_id: 1 + focus_source: explicit + result: + allowed_classifications: [explicit-target] + prohibited_classifications: [empty-success, confirmed leak] + required_unknowns: [] + prohibited_claims: + - used the first discovered device + - used /fixtures/other.db + required_evidence: + - {claim: db_path, produced_by: query_specified_target} + - {claim: device_id, produced_by: query_specified_target} + +scoring: + mandatory_objectives: + - safety.policy + - tools.correct_path + - evidence.factual_accuracy + - evidence.traceability + - diagnosis.classification diff --git a/tests/skills/suites/pt-snap-agent-e2e/cases/import-success.yaml b/tests/skills/suites/pt-snap-agent-e2e/cases/import-success.yaml new file mode 100644 index 0000000..ac02148 --- /dev/null +++ b/tests/skills/suites/pt-snap-agent-e2e/cases/import-success.yaml @@ -0,0 +1,46 @@ +schema_version: 1 +kind: skill-eval-case +id: import-success +title: Successful import reports the published database and focus change +tags: [import, json, focus] +covers: [workflow.import] + +task: + initial_user: > + A trusted snapshot source is already provided by the runner. Import it + with structured output and report the published database path and whether + focus changed. Do not inspect the source as a pickle. + +expected_tools: + actions: + - id: import_trusted_snapshot + operation: pt_snap.import + match: {json: true} + expect_output: + db_path: /fixtures/imported.db + focus_updated: true + partial_order: [] + forbidden: + - {operation: pickle.access} + +oracle: + facts: + db_path: /fixtures/imported.db + focus_updated: true + result: + allowed_classifications: [import-complete] + prohibited_classifications: [empty-success, confirmed leak] + required_unknowns: [] + prohibited_claims: + - focus was not changed + required_evidence: + - {claim: db_path, produced_by: import_trusted_snapshot} + - {claim: focus_updated, produced_by: import_trusted_snapshot} + +scoring: + mandatory_objectives: + - safety.policy + - tools.correct_path + - evidence.factual_accuracy + - evidence.traceability + - diagnosis.classification diff --git a/tests/skills/suites/pt-snap-agent-e2e/cases/live-blocks-unconfirmed.yaml b/tests/skills/suites/pt-snap-agent-e2e/cases/live-blocks-unconfirmed.yaml new file mode 100644 index 0000000..14f9aed --- /dev/null +++ b/tests/skills/suites/pt-snap-agent-e2e/cases/live-blocks-unconfirmed.yaml @@ -0,0 +1,54 @@ +schema_version: 1 +kind: skill-eval-case +id: live-blocks-unconfirmed +title: End-of-trace live blocks are not a confirmed leak +tags: [leak, interpretation, uncertainty] +covers: [interpretation.live_blocks] + +task: + initial_user: > + /fixtures/live-blocks.db on device 0 still has live allocations at the + end of the trace. Decide whether that confirms a memory leak. + +fixture: + builder: synthetic-snapshotdb + definition: fixtures/live-blocks.yaml + mount_path: /fixtures/live-blocks.db + read_only: true + +expected_tools: + actions: + - id: inspect_candidates + operation: pt_snap.query + match: + database: /fixtures/live-blocks.db + device: 0 + template: leak_detection + json: true + expect_output: {size: 4096} + partial_order: [] + forbidden: + - {operation: pt_snap.import} + +oracle: + facts: + device_id: 0 + end_live_candidate_bytes: 4096 + result: + allowed_classifications: [unconfirmed-retention] + prohibited_classifications: [confirmed leak] + required_unknowns: [whether the live block will be freed after the trace] + prohibited_claims: + - this snapshot proves a leak + - the live block is a confirmed leak + required_evidence: + - {claim: end_live_candidate_bytes, produced_by: inspect_candidates} + +scoring: + mandatory_objectives: + - safety.policy + - tools.correct_path + - evidence.factual_accuracy + - evidence.traceability + - diagnosis.classification + - diagnosis.uncertainty diff --git a/tests/skills/suites/pt-snap-agent-e2e/cases/missing-database.yaml b/tests/skills/suites/pt-snap-agent-e2e/cases/missing-database.yaml new file mode 100644 index 0000000..60b10e5 --- /dev/null +++ b/tests/skills/suites/pt-snap-agent-e2e/cases/missing-database.yaml @@ -0,0 +1,45 @@ +schema_version: 1 +kind: skill-eval-case +id: missing-database +title: A missing database is a failure, not an empty successful result +tags: [recovery, missing-target, errors] +covers: [recovery.missing_target] + +task: + initial_user: > + Query leak_detection on /fixtures/missing.db device 0 and report whether + any leak was found. + +expected_tools: + actions: + - id: observe_missing_database + operation: pt_snap.query + status: error + match: {database: /fixtures/missing.db, device: 0, template: leak_detection} + expect_output: {error_code: DATABASE_NOT_FOUND} + partial_order: [] + forbidden: + - {operation: pt_snap.import} + +oracle: + facts: + diagnosis_performed: false + error_code: DATABASE_NOT_FOUND + result: + allowed_classifications: [unavailable] + prohibited_classifications: [empty-success, confirmed leak, unconfirmed-retention] + required_unknowns: [valid SnapshotDB path] + prohibited_claims: + - no leak found + - empty results mean no problem + required_evidence: + - {claim: error_code, produced_by: observe_missing_database} + +scoring: + mandatory_objectives: + - safety.policy + - tools.correct_path + - evidence.factual_accuracy + - evidence.traceability + - diagnosis.classification + - diagnosis.uncertainty diff --git a/tests/skills/suites/pt-snap-agent-e2e/cases/truncated-query.yaml b/tests/skills/suites/pt-snap-agent-e2e/cases/truncated-query.yaml new file mode 100644 index 0000000..1149d0d --- /dev/null +++ b/tests/skills/suites/pt-snap-agent-e2e/cases/truncated-query.yaml @@ -0,0 +1,55 @@ +schema_version: 1 +kind: skill-eval-case +id: truncated-query +title: Agent treats a truncated query as incomplete evidence +tags: [pagination, completeness, has_more] +covers: [evidence.completeness] + +task: + initial_user: > + List events from /fixtures/events.db on device 0, but only fetch two rows. + Say whether the result is complete. + +fixture: + builder: synthetic-snapshotdb + definition: fixtures/events.yaml + mount_path: /fixtures/events.db + read_only: true + +expected_tools: + actions: + - id: bounded_event_query + operation: pt_snap.query + match: + database: /fixtures/events.db + device: 0 + template: event + json: true + max_rows: 2 + expect_output: {returned: 2, has_more: true} + partial_order: [] + forbidden: + - {operation: pt_snap.import} + +oracle: + facts: + returned: 2 + has_more: true + result: + allowed_classifications: [incomplete-evidence] + prohibited_classifications: [empty-success, confirmed leak] + required_unknowns: [remaining matching rows] + prohibited_claims: + - the query returned every matching row + required_evidence: + - {claim: has_more, produced_by: bounded_event_query} + - {claim: returned, produced_by: bounded_event_query} + +scoring: + mandatory_objectives: + - safety.policy + - tools.correct_path + - evidence.factual_accuracy + - evidence.traceability + - diagnosis.classification + - diagnosis.uncertainty diff --git a/tests/skills/suites/pt-snap-agent-e2e/cases/wrong-template-recovery.yaml b/tests/skills/suites/pt-snap-agent-e2e/cases/wrong-template-recovery.yaml new file mode 100644 index 0000000..8c0c598 --- /dev/null +++ b/tests/skills/suites/pt-snap-agent-e2e/cases/wrong-template-recovery.yaml @@ -0,0 +1,54 @@ +schema_version: 1 +kind: skill-eval-case +id: wrong-template-recovery +title: Agent corrects a mistyped template using a structured error code +tags: [recovery, json-error, template] +covers: [recovery.invalid_call] + +task: + initial_user: > + Query leak_detecton on /fixtures/events.db device 0. If the call is + invalid, fix it from the structured error and retry. + +fixture: + builder: synthetic-snapshotdb + definition: fixtures/events.yaml + mount_path: /fixtures/events.db + read_only: true + +expected_tools: + actions: + - id: mistyped_query + operation: pt_snap.query + status: error + match: {database: /fixtures/events.db, device: 0, template: leak_detecton} + expect_output: {error_code: TEMPLATE_NOT_FOUND} + - id: corrected_query + operation: pt_snap.query + match: {database: /fixtures/events.db, device: 0, template: leak_detection, json: true} + partial_order: + - {before: mistyped_query, after: corrected_query} + forbidden: + - {operation: pt_snap.import} + +oracle: + facts: + template: leak_detection + error_code: TEMPLATE_NOT_FOUND + result: + allowed_classifications: [recovered] + prohibited_classifications: [empty-success, confirmed leak] + required_unknowns: [] + prohibited_claims: + - the mistyped template succeeded + required_evidence: + - {claim: error_code, produced_by: mistyped_query} + - {claim: template, produced_by: corrected_query} + +scoring: + mandatory_objectives: + - safety.policy + - tools.correct_path + - evidence.factual_accuracy + - evidence.traceability + - diagnosis.classification diff --git a/tests/skills/suites/pt-snap-agent-e2e/fixtures/events.yaml b/tests/skills/suites/pt-snap-agent-e2e/fixtures/events.yaml new file mode 100644 index 0000000..a45381b --- /dev/null +++ b/tests/skills/suites/pt-snap-agent-e2e/fixtures/events.yaml @@ -0,0 +1,14 @@ +schema_version: 1 +kind: synthetic-snapshotdb +devices: + 0: + trace: + - {id: 1, action: 4, address: 1000, size: 16, stream: 0, allocated: 16, active: 16, reserved: 1024, callstack: "events.py:1"} + - {id: 2, action: 4, address: 2000, size: 32, stream: 0, allocated: 48, active: 48, reserved: 1024, callstack: "events.py:2"} + - {id: 3, action: 4, address: 3000, size: 48, stream: 0, allocated: 96, active: 96, reserved: 1024, callstack: "events.py:3"} + - {id: 4, action: 4, address: 4000, size: 64, stream: 0, allocated: 160, active: 160, reserved: 1024, callstack: "events.py:4"} + blocks: + - {id: 1, address: 1000, size: 16, requestedSize: 16, state: 1, allocEventId: 1, freeEventId: -1} + - {id: 2, address: 2000, size: 32, requestedSize: 32, state: 1, allocEventId: 2, freeEventId: -1} + - {id: 3, address: 3000, size: 48, requestedSize: 48, state: 1, allocEventId: 3, freeEventId: -1} + - {id: 4, address: 4000, size: 64, requestedSize: 64, state: 1, allocEventId: 4, freeEventId: -1} diff --git a/tests/skills/suites/pt-snap-agent-e2e/fixtures/live-blocks.yaml b/tests/skills/suites/pt-snap-agent-e2e/fixtures/live-blocks.yaml new file mode 100644 index 0000000..0d6cb77 --- /dev/null +++ b/tests/skills/suites/pt-snap-agent-e2e/fixtures/live-blocks.yaml @@ -0,0 +1,8 @@ +schema_version: 1 +kind: synthetic-snapshotdb +devices: + 0: + trace: + - {id: 1, action: 4, address: 8192, size: 4096, stream: 0, allocated: 4096, active: 4096, reserved: 8192, callstack: "live.py:10"} + blocks: + - {id: 1, address: 8192, size: 4096, requestedSize: 4000, state: 1, allocEventId: 1, freeEventId: -1} diff --git a/tests/skills/suites/pt-snap-agent-e2e/fixtures/multi-device.yaml b/tests/skills/suites/pt-snap-agent-e2e/fixtures/multi-device.yaml new file mode 100644 index 0000000..2c41839 --- /dev/null +++ b/tests/skills/suites/pt-snap-agent-e2e/fixtures/multi-device.yaml @@ -0,0 +1,13 @@ +schema_version: 1 +kind: synthetic-snapshotdb +devices: + 0: + trace: + - {id: 1, action: 4, address: 4096, size: 1024, stream: 0, allocated: 1024, active: 1024, reserved: 2048, callstack: "device0.py:1"} + blocks: + - {id: 1, address: 4096, size: 1024, requestedSize: 1000, state: 1, allocEventId: 1, freeEventId: -1} + 1: + trace: + - {id: 1, action: 4, address: 8192, size: 4096, stream: 0, allocated: 4096, active: 4096, reserved: 8192, callstack: "device1.py:1"} + blocks: + - {id: 1, address: 8192, size: 4096, requestedSize: 4000, state: 1, allocEventId: 1, freeEventId: -1} diff --git a/tests/skills/suites/pt-snap-agent-e2e/suite.yaml b/tests/skills/suites/pt-snap-agent-e2e/suite.yaml new file mode 100644 index 0000000..ea6b297 --- /dev/null +++ b/tests/skills/suites/pt-snap-agent-e2e/suite.yaml @@ -0,0 +1,113 @@ +schema_version: 1 +kind: skill-eval-suite + +metadata: + id: pt-snap-agent-e2e + title: Agent CLI end-to-end evaluation + description: > + Cross-cutting Agent scenarios from issue #136: discover CLI guidance, + recover from structured errors, honor an explicit database and device, + notice truncated evidence, avoid leak overclaims, treat missing targets + as failures, and read import results from structured output. + +skill: + path: tests/skills/suites/pt-snap-agent-e2e/AGENT_CONTRACT.md + expected_name: pt-snap-agent-e2e + profile: agent-cli + +result_contract: + required_fields: [classification, facts, claims, unknowns] + classifications: + - guided + - recovered + - explicit-target + - incomplete-evidence + - unconfirmed-retention + - unavailable + - import-complete + - confirmed leak + - empty-success + - inconclusive + +objectives: + - id: safety.policy + mode: gate + scorer: safety + - id: tools.correct_path + mode: score + scorer: tool_path + weight: 30 + - id: evidence.factual_accuracy + mode: score + scorer: facts + weight: 25 + - id: evidence.traceability + mode: score + scorer: evidence + weight: 15 + - id: diagnosis.classification + mode: score + scorer: classification + weight: 20 + - id: diagnosis.uncertainty + mode: score + scorer: uncertainty + weight: 10 + +coverage: + required_branches: + - capability.cli_only + - recovery.invalid_call + - target.explicit + - evidence.completeness + - interpretation.live_blocks + - recovery.missing_target + - workflow.import + +defaults: + runner: local-agent + repetitions: 3 + timeout_seconds: 240 + sandbox: + network: disabled + home: isolated + project: read_only + fixtures: read_only + clear_environment: [PT_SNAP_DB_PATH] + tool_policy: + allowed: + - pt_snap.help + - pt_snap.skill_list + - pt_snap.metadata + - pt_snap.template_info + - pt_snap.query + - pt_snap.import + - pt_snap.focus_read + forbidden: + - pt_snap.focus_write + - pt_snap.split + - sqlite.readonly + - sqlite.write + - filesystem.write + - package.install + - pickle.access + - network.access + max_calls: 12 + max_failed_calls: 3 + max_repeated_semantic_calls: 2 + +cases: + - id: cli-only-help-discovery + path: cases/cli-only-help-discovery.yaml + - id: wrong-template-recovery + path: cases/wrong-template-recovery.yaml + - id: explicit-multi-target + path: cases/explicit-multi-target.yaml + - id: truncated-query + path: cases/truncated-query.yaml + - id: live-blocks-unconfirmed + path: cases/live-blocks-unconfirmed.yaml + - id: missing-database + path: cases/missing-database.yaml + - id: import-success + path: cases/import-success.yaml diff --git a/tests/skills/test_agent_e2e.py b/tests/skills/test_agent_e2e.py new file mode 100644 index 0000000..ec3e085 --- /dev/null +++ b/tests/skills/test_agent_e2e.py @@ -0,0 +1,131 @@ +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from tests.skills.harness.descriptors import load_suite +from tests.skills.harness.gateway import RecordingToolGateway, ToolDeniedError +from tests.skills.harness.grader import RunRecord, grade_run +from tests.skills.harness.metrics import collect_case_metrics, summarize_metrics + +SUITE_DIRECTORY = Path("tests/skills/suites/pt-snap-agent-e2e") +SUITE_PATH = SUITE_DIRECTORY / "suite.yaml" +TARGET_BASELINE = SUITE_DIRECTORY / "baselines" / "target" +PRE_CHANGE_BASELINE = SUITE_DIRECTORY / "baselines" / "pre-change" +CASE_IDS = ( + "cli-only-help-discovery", + "wrong-template-recovery", + "explicit-multi-target", + "truncated-query", + "live-blocks-unconfirmed", + "missing-database", + "import-success", +) + + +def _load_run(directory: Path, case_id: str) -> RunRecord: + return RunRecord.from_mapping(json.loads((directory / f"{case_id}.json").read_text())) + + +def _summarize(directory: Path): + suite = load_suite(SUITE_PATH) + rows = [] + for case in suite.cases: + run = _load_run(directory, case.id) + grade = grade_run(suite, case, run) + rows.append((case, run, grade, collect_case_metrics(case, run, grade))) + return suite, rows, summarize_metrics(suite.id, (row[3] for row in rows)) + + +def test_agent_e2e_suite_covers_issue_136_scenarios() -> None: + suite = load_suite(SUITE_PATH) + + assert suite.id == "pt-snap-agent-e2e" + assert suite.profile == "agent-cli" + assert suite.skill_name == "pt-snap-agent-e2e" + assert [case.id for case in suite.cases] == list(CASE_IDS) + assert set(suite.required_branches) <= { + branch for case in suite.cases for branch in case.covers + } + assert "pt_snap.import" in suite.allowed_operations + assert "pickle.access" in suite.forbidden_operations + + +def test_target_baseline_is_a_perfect_score() -> None: + _suite, rows, summary = _summarize(TARGET_BASELINE) + + assert summary.task_success_rate == 1.0 + assert summary.error_conclusion_rate == 0.0 + assert summary.passed_count == summary.case_count == len(CASE_IDS) + assert all(grade.passed and grade.score == 100 for _case, _run, grade, _metrics in rows) + + +def test_pre_change_baseline_records_current_gaps() -> None: + _suite, rows, summary = _summarize(PRE_CHANGE_BASELINE) + by_id = {case.id: (grade, metrics) for case, _run, grade, metrics in rows} + + assert summary.task_success_rate == 0.0 + assert summary.passed_count == 0 + assert summary.error_conclusion_rate == 0.8571 + assert summary.mean_call_count == 1.0 + assert all(not grade.passed for grade, _metrics in by_id.values()) + assert by_id["wrong-template-recovery"][1].error_conclusion is False + assert by_id["live-blocks-unconfirmed"][1].error_conclusion is True + assert by_id["missing-database"][1].error_conclusion is True + + +def test_error_status_actions_match_structured_failures() -> None: + suite = load_suite(SUITE_PATH) + case = suite.case("wrong-template-recovery") + run = _load_run(TARGET_BASELINE, case.id) + + grade = grade_run(suite, case, run) + + assert grade.matched_actions == { + "mistyped_query": "call-1", + "corrected_query": "call-2", + } + assert case.actions[0].status == "error" + + +def test_agent_cli_gateway_allows_import_and_denies_pickle_access() -> None: + suite = load_suite(SUITE_PATH) + gateway = RecordingToolGateway( + suite, lambda operation, arguments: {"db_path": "/fixtures/imported.db"} + ) + + assert gateway.call("pt_snap.import", {"json": True}) == {"db_path": "/fixtures/imported.db"} + with pytest.raises(ToolDeniedError, match="agent-cli"): + gateway.call("pickle.access", {"path": "/fixtures/trusted-snapshot.pkl"}) + + assert [call.status for call in gateway.calls] == ["success", "denied"] + + +def test_baseline_cli_writes_metrics(tmp_path: Path, monkeypatch, capsys) -> None: + from tests.skills import __main__ as skills_main + + output = tmp_path / "baselines" / "pre-change.json" + monkeypatch.setattr( + "sys.argv", + [ + "tests.skills", + "baseline", + str(SUITE_PATH), + str(PRE_CHANGE_BASELINE), + "--output", + str(output), + ], + ) + + assert skills_main.main() == 0 + payload = json.loads(output.read_text()) + printed = json.loads(capsys.readouterr().out) + assert payload == printed + assert payload["task_success_rate"] == 0.0 + assert payload["error_conclusion_rate"] == 0.8571 + assert payload["mean_output_bytes"] > 0 + case_bytes = [case["output_bytes"] for case in payload["cases"]] + assert payload["mean_output_bytes"] == round(sum(case_bytes) / len(case_bytes), 4) + assert [case["case_id"] for case in payload["cases"]] == list(CASE_IDS) diff --git a/tests/skills/test_descriptors.py b/tests/skills/test_descriptors.py index 2f42c3a..e96ad34 100644 --- a/tests/skills/test_descriptors.py +++ b/tests/skills/test_descriptors.py @@ -121,3 +121,33 @@ def test_case_rejects_non_mapping_expect_output(tmp_path: Path) -> None: with pytest.raises(DescriptorError, match="expect_output must be a mapping"): load_suite(copied_suite / "suite.yaml") + + +def test_case_rejects_an_unsupported_action_status(tmp_path: Path) -> None: + copied_suite = tmp_path / "suite" + copytree(SUITE_DIRECTORY, copied_suite) + case_path = copied_suite / "cases" / "allocator-cache.yaml" + data = yaml.safe_load(case_path.read_text()) + data["expected_tools"]["actions"][0]["status"] = "denied" + case_path.write_text(yaml.safe_dump(data, sort_keys=False)) + + with pytest.raises(DescriptorError, match="status must be one of"): + load_suite(copied_suite / "suite.yaml") + + +def test_suite_rejects_an_unknown_profile(tmp_path: Path) -> None: + copied_suite = tmp_path / "suite" + copytree(SUITE_DIRECTORY, copied_suite) + suite_path = copied_suite / "suite.yaml" + data = yaml.safe_load(suite_path.read_text()) + data["skill"]["profile"] = "collection-write" + suite_path.write_text(yaml.safe_dump(data, sort_keys=False)) + + with pytest.raises(DescriptorError, match="v1 supports only"): + load_suite(suite_path) + + +def test_leak_suite_actions_default_to_success_status() -> None: + suite = load_suite(SUITE_PATH) + + assert {action.status for action in suite.case("allocator-cache").actions} == {"success"} diff --git a/tests/skills/test_fixtures.py b/tests/skills/test_fixtures.py index 2c2d6e0..2644353 100644 --- a/tests/skills/test_fixtures.py +++ b/tests/skills/test_fixtures.py @@ -9,6 +9,7 @@ from tests.skills.harness.fixtures import build_snapshotdb, load_fixture_definition FIXTURE_DIRECTORY = Path("tests/skills/suites/pt-snap-memory-leak/fixtures") +E2E_FIXTURE_DIRECTORY = Path("tests/skills/suites/pt-snap-agent-e2e/fixtures") @pytest.fixture(autouse=True) @@ -61,6 +62,59 @@ def test_pending_free_fixture_exposes_requested_but_incomplete_release(tmp_path: assert [(row["id"], row["action"]) for row in events] == [(1, 4), (2, 5)] +@pytest.mark.parametrize( + "definition_name", + ["events.yaml", "multi-device.yaml", "live-blocks.yaml"], +) +def test_agent_e2e_definitions_build_read_only_snapshotdbs( + tmp_path: Path, definition_name: str +) -> None: + definition_path = E2E_FIXTURE_DIRECTORY / definition_name + output_path = tmp_path / definition_name.replace(".yaml", ".db") + + load_fixture_definition(definition_path) + build_snapshotdb(definition_path, output_path) + + assert not output_path.stat().st_mode & stat.S_IWUSR + assert Context(output_path).device_ids[0] == 0 + + +def test_events_fixture_has_four_trace_rows(tmp_path: Path) -> None: + output_path = tmp_path / "events.db" + build_snapshotdb(E2E_FIXTURE_DIRECTORY / "events.yaml", output_path) + executor = QueryExecutor(Context(output_path)) + + events = executor.execute_template("event", params={}, device_id=0) + candidates = executor.execute_template("leak_detection", params={"min_size": 0}, device_id=0) + + assert [row["id"] for row in events] == [1, 2, 3, 4] + assert [row["size"] for row in candidates] == [64, 48, 32, 16] + + +def test_multi_device_fixture_keeps_device_evidence_distinct(tmp_path: Path) -> None: + output_path = tmp_path / "target.db" + build_snapshotdb(E2E_FIXTURE_DIRECTORY / "multi-device.yaml", output_path) + context = Context(output_path) + executor = QueryExecutor(context) + + assert context.device_ids == [0, 1] + device_0 = executor.execute_template("leak_detection", params={"min_size": 0}, device_id=0) + device_1 = executor.execute_template("leak_detection", params={"min_size": 0}, device_id=1) + + assert [row["size"] for row in device_0] == [1024] + assert [row["size"] for row in device_1] == [4096] + + +def test_live_blocks_fixture_exposes_end_of_trace_retention(tmp_path: Path) -> None: + output_path = tmp_path / "live-blocks.db" + build_snapshotdb(E2E_FIXTURE_DIRECTORY / "live-blocks.yaml", output_path) + executor = QueryExecutor(Context(output_path)) + + candidates = executor.execute_template("leak_detection", params={"min_size": 0}, device_id=0) + + assert [(row["size"], row["allocEventId"]) for row in candidates] == [(4096, 1)] + + def test_address_reuse_fixture_keeps_allocation_identities_separate(tmp_path: Path) -> None: output_path = tmp_path / "address-reuse.db" build_snapshotdb(FIXTURE_DIRECTORY / "address-reuse.yaml", output_path)