Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ htmlcov/
.codex/*
.opencode/*
.tmp/*
.skill-evals/
openspec/
specs/

Expand Down
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ templates under category subdirectories are included by
| Snapshot import or metadata | `src/pt_snap_cli/core/import_service.py`, `src/pt_snap_cli/core/import_metadata.py`, `src/pt_snap_cli/core/snapshot_import_backend.py` | `tests/core/test_import_*.py`, `tests/test_snapshot_db.py` |
| Snapshot splitting or replay | `src/pt_snap_cli/core/split_service.py`, `src/pt_snap_cli/snapshot/` | `tests/core/test_split_service.py`, `tests/snapshot/` |
| Reports | `src/pt_snap_cli/core/report_service.py`, report commands in `src/pt_snap_cli/cli.py` | `tests/core/test_report_service.py`, report cases in `tests/test_cli.py` |
| Agent setup or diagnostic skills | `skills/` and the referenced CLI/query surfaces | `tests/test_setup_skill.py`, `tests/test_memory_leak_skill.py`, `tests/test_memory_peak_breakdown_skill.py`, `tests/test_memory_fragmentation_skill.py` |
| Agent setup, diagnostics, or skill evaluation | `skills/`, `tests/skills/`, and the referenced CLI/query surfaces | `tests/skills/` |
| Packaging or release | `pyproject.toml`, `.github/workflows/` | `tests/test_package.py`, `tests/test_release_workflow.py` |
| Executable fixtures | `tests/fixtures/snapshots/` | `tests/test_fixture_provenance.py` before any deserializing suite |

Expand Down
12 changes: 4 additions & 8 deletions skills/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,11 +30,7 @@ SnapshotDB data without writes.
document why a template is insufficient and enforce read-only access.

## Focused Tests
- Run `pytest tests/test_setup_skill.py` after setup-skill changes. The current executable test covers active-interpreter path preservation; review the remaining approval and reporting instructions statically.
- Run `pytest tests/test_memory_leak_skill.py` after memory-leak skill changes;
it checks current template references, conservative classification, and the
read-only diagnostic boundary.
- Run `pytest tests/test_memory_peak_breakdown_skill.py` after peak-breakdown skill changes; it locks command/template references, range fallback, and attribution caveats.
- Run `pytest tests/test_memory_fragmentation_skill.py` after fragmentation-skill
changes; it checks current command/template references, read-only boundaries,
runtime segment coverage, and conservative classifications.
- Run `pytest tests/skills/test_setup_contract.py` after setup-skill changes. The current executable test covers active-interpreter path preservation; review the remaining approval and reporting instructions statically.
- Run `pytest tests/skills` after diagnostic-skill changes. Static contracts
check current surfaces and safety boundaries; suite/case evaluations cover
decision paths, structured evidence, and tool-call policy.
6 changes: 2 additions & 4 deletions tests/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,13 +20,9 @@
| `test_fixture_provenance.py` | Non-deserializing coverage, SHA-256, size, and Git LFS pointer validation for executable fixtures. |
| `test_governance.py` | Snapshot provenance and repository governance contracts. |
| `test_mcp_server.py` | MCP server tool/resource/prompt behavior tests. |
| `test_memory_leak_skill.py` | Verifies the memory-leak skill's current template references, conservative classifications, and read-only boundaries. |
| `test_memory_fragmentation_skill.py` | Verifies the fragmentation skill's current command/template references, runtime segment coverage, conservative classifications, and read-only boundaries. |
| `test_models.py` | Package-level model behavior tests. |
| `test_package.py` | Package metadata/import/version tests. |
| `test_release_workflow.py` | Release workflow and package publication contract tests. |
| `test_memory_peak_breakdown_skill.py` | Static contracts for the read-only peak-memory diagnostic skill and its attribution limits. |
| `test_setup_skill.py` | Verifies that the setup skill preserves the selected active interpreter path. |
| `test_snapshot_analyzer_cache.py`, `test_query_cache_perf.py` | Analyzer cache invalidation and repeated-query connection reuse contracts. |
| `test_snapshot_db.py` | SnapshotDb write pragmas and query-index creation. |
| `test_baseline_import.py` | Import benchmark metric parsing and platform RSS normalization. |
Expand All @@ -39,6 +35,7 @@
| `models/` | Domain model and enum unit tests governed by this guide. |
| `query/` | Query builder, config, executor, mapper, registry, and condition tests (see `query/AGENTS.md`). |
| `snapshot/` | Representation, replay, database adaptor, and slicing runtime tests (see `snapshot/AGENTS.md`). |
| `skills/` | Static skill contracts and local-first suite/case evaluation harness (see `skills/AGENTS.md`). |

## For AI Agents

Expand All @@ -47,6 +44,7 @@
- Use temporary directories and SQLite fixtures instead of relying on local `.pt-snap/focus.json` or real snapshot databases.
- Keep tests deterministic across working directories and user machines.
- Isolate CWD, `PT_SNAP_DB_PATH`, and `Path.home()` in tests that resolve or persist focus.
- Keep live agent runs out of normal pytest; skill evaluation definitions and deterministic graders live under `tests/skills/`.

### Testing Requirements
- Run `pytest` for the full suite.
Expand Down
28 changes: 28 additions & 0 deletions tests/skills/AGENTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
<!-- Parent: ../AGENTS.md -->

# skill evaluations

## Purpose
`tests/skills` contains static skill contracts and a local-first evaluation
harness for versioned suite/case descriptors, synthetic SnapshotDB scenarios,
tool-call traces, deterministic grading, and runner adapters.

## Scope
| Path | Responsibility |
| --- | --- |
| `schemas/` | Human-readable v1 suite and case descriptor contracts. |
| `harness/` | Strict descriptor loading, synthetic fixture construction, trace grading, runner protocols, and local artifacts. |
| `suites/` | Reviewed skill-specific objectives, decision branches, cases, and declarative SQLite fixtures. |
| `README.md` | Descriptor, gateway, run-record, grading, and local command reference. |
| `test_*_contract.py` | Static and executable contracts for shipped `SKILL.md` files. |

## Invariants
- Keep evaluation definitions declarative; do not embed shell, Python, Jinja, or arbitrary expressions in YAML.
- Diagnostic suites use generated SnapshotDB files only. Never expose, import, or deserialize pickle fixtures in an agent run.
- Treat forbidden operations as hard failures and grade allowed tool calls by normalized semantics rather than shell text.
- Keep live model execution local and explicit. Normal pytest coverage must not require network access, provider credentials, or a model runtime.
- Write generated transcripts, databases, scores, and reports only under temporary directories or the ignored `.skill-evals/` directory.

## Focused Tests
- Run `pytest tests/skills` for harness, descriptor, fixture, grader, or skill contract changes.
- Run `python -m tests.skills validate tests/skills/suites/<skill>/suite.yaml` after editing a suite or case descriptor.
91 changes: 91 additions & 0 deletions tests/skills/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
# Local Skill Evaluation

`tests/skills` evaluates repository agent skills without requiring a model,
network access, or GitHub CI. Live-agent adapters can use the same descriptors,
tool gateway, run record, and deterministic grader from a fresh local session.

## Descriptor Model

Each suite has one `suite.yaml` and explicit case references under
`suites/<skill>/`. The versioned contracts are documented in `schemas/` and
enforced by `harness/descriptors.py` with unknown fields rejected.

- A suite identifies the `SKILL.md`, result classifications, scored objectives,
required decision branches, sandbox defaults, semantic tool policy, and cases.
Runner defaults (`runner`, `repetitions`, `timeout_seconds`, `sandbox`) are
validated and exposed as typed values on the loaded suite so adapters can
enforce them.
- A case identifies its prompt, optional synthetic SnapshotDB, covered branches,
required tool actions, partial ordering, forbidden actions, oracle facts,
classification bounds, unknowns, and claim-to-tool evidence links. Actions may
declare `expect_output`, a mapping that must appear in the matched call's
recorded output before the action counts as matched. Cases may cap the total
call count with `expected_tools.max_calls`; zero enforces refusal-only cases.
- Descriptor paths are repository-relative and cannot contain `..`. Fixture
mount paths must be normalized absolute POSIX paths under `/fixtures/`.
- Diagnostic fixtures are declarative SQLite databases. Pickle inputs are never
materialized or exposed to a diagnostic runner.

Validate a suite locally:

```bash
python -m tests.skills validate tests/skills/suites/pt-snap-memory-leak/suite.yaml
```

## Tool Gateway

Agent adapters should expose only semantic operations through
`RecordingToolGateway`. The gateway records successful, failed, and denied
attempts and enforces the suite allowlist and call budget. Adapters remain
responsible for process-level isolation: a fresh temporary working directory,
an isolated `HOME`, no network, a read-only project, read-only fixture mounts,
and cleared focus environment variables.

The gateway operation names are transport-independent. An adapter may implement
them with a CLI wrapper, MCP server, or another local agent tool, but raw command
spelling is not part of the grading contract.

## Run Record

A runner submits JSON with the following shape:

```json
{
"tool_calls": [
{
"id": "call-1",
"operation": "pt_snap.metadata",
"arguments": {"database": "/fixtures/cache.db", "json": true},
"status": "success",
"output": {"status": "unavailable", "reason": "metadata_missing", "metadata": null}
}
],
"result": {
"classification": "allocator/cache effect",
"facts": {"device_id": 0},
"claims": [
{"id": "device_id", "evidence_call_ids": ["call-1"]}
],
"unknowns": ["repeated-capture evidence"]
},
"final_response": "Evidence-backed user-facing response"
}
```

Grade a recorded run and optionally write local artifacts:

```bash
python -m tests.skills grade \
tests/skills/suites/pt-snap-memory-leak/suite.yaml \
allocator-cache run.json --output .skill-evals/runs/example
```

Safety objectives are hard gates. Scored objectives use normalized operation
arguments, required output evidence, partial ordering, tool budgets (suite-wide
and per case), oracle facts, classifications, required unknowns, and evidence
call IDs. Runs that omit required result fields or submit malformed result
shapes also hard-fail. Formatting and prose style receive no deterministic
score.

Generated transcripts and reports belong under `.skill-evals/`, which is
ignored by Git. Normal `pytest` runs never invoke a live model.
1 change: 1 addition & 0 deletions tests/skills/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
"""Local-first evaluation support for repository agent skills."""
61 changes: 61 additions & 0 deletions tests/skills/__main__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
from __future__ import annotations

import argparse
import json
from pathlib import Path

from .harness.artifacts import write_run_artifacts
from .harness.descriptors import load_suite
from .harness.grader import RunRecord, grade_run


def _validate(args: argparse.Namespace) -> int:
suite = load_suite(args.suite)
covered = sorted({branch for case in suite.cases for branch in case.covers})
print(
json.dumps(
{
"suite": suite.id,
"skill": suite.skill_name,
"cases": [case.id for case in suite.cases],
"covered_branches": covered,
},
indent=2,
sort_keys=True,
)
)
return 0


def _grade(args: argparse.Namespace) -> int:
suite = load_suite(args.suite)
case = suite.case(args.case)
run = RunRecord.from_mapping(json.loads(args.run.read_text()))
grade = grade_run(suite, case, run)
if args.output is not None:
write_run_artifacts(args.output, suite, case, run, grade)
print(json.dumps(grade.to_mapping(), indent=2, sort_keys=True))
return 0 if grade.passed else 1


def main() -> int:
parser = argparse.ArgumentParser(description="Validate and grade local skill evaluations")
subparsers = parser.add_subparsers(dest="command", required=True)

validate_parser = subparsers.add_parser("validate", help="validate a suite and its cases")
validate_parser.add_argument("suite", type=Path)
validate_parser.set_defaults(handler=_validate)

grade_parser = subparsers.add_parser("grade", help="grade a recorded agent run")
grade_parser.add_argument("suite", type=Path)
grade_parser.add_argument("case")
grade_parser.add_argument("run", type=Path)
grade_parser.add_argument("--output", type=Path)
grade_parser.set_defaults(handler=_grade)

args = parser.parse_args()
return args.handler(args)


if __name__ == "__main__":
raise SystemExit(main())
19 changes: 19 additions & 0 deletions tests/skills/harness/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
"""Descriptor, fixture, runner, and grading primitives for skill evaluations."""

from .descriptors import DescriptorError, EvalCase, EvalSuite, SandboxPolicy, load_suite
from .gateway import RecordingToolGateway, ToolDeniedError
from .grader import GradeResult, RunRecord, ToolCall, grade_run

__all__ = [
"DescriptorError",
"EvalCase",
"EvalSuite",
"GradeResult",
"RecordingToolGateway",
"RunRecord",
"SandboxPolicy",
"ToolCall",
"ToolDeniedError",
"grade_run",
"load_suite",
]
64 changes: 64 additions & 0 deletions tests/skills/harness/artifacts.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
from __future__ import annotations

import hashlib
import json
from pathlib import Path

from .descriptors import REPO_ROOT, EvalCase, EvalSuite
from .grader import GradeResult, RunRecord


def _sha256(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as stream:
for chunk in iter(lambda: stream.read(1024 * 1024), b""):
digest.update(chunk)
return digest.hexdigest()


def write_run_artifacts(
output_directory: Path,
suite: EvalSuite,
case: EvalCase,
run: RunRecord,
grade: GradeResult,
) -> None:
output_directory.mkdir(parents=True, exist_ok=False)
manifest = {
"suite_id": suite.id,
"case_id": case.id,
"skill_path": str(suite.skill_path.relative_to(REPO_ROOT)),
"skill_sha256": _sha256(suite.skill_path),
"suite_sha256": _sha256(suite.path),
"case_sha256": _sha256(case.path),
}
(output_directory / "manifest.json").write_text(
json.dumps(manifest, indent=2, sort_keys=True) + "\n"
)
with (output_directory / "trace.jsonl").open("w") as stream:
for call in run.tool_calls:
stream.write(
json.dumps(
{
"id": call.id,
"operation": call.operation,
"arguments": call.arguments,
"status": call.status,
"output": call.output,
"error": call.error,
},
sort_keys=True,
)
+ "\n"
)
(output_directory / "outcome.json").write_text(
json.dumps(
{"result": run.result, "final_response": run.final_response},
indent=2,
sort_keys=True,
)
+ "\n"
)
(output_directory / "score.json").write_text(
json.dumps(grade.to_mapping(), indent=2, sort_keys=True) + "\n"
)
Loading
Loading