Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions skills/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,8 +47,8 @@ restarted.
## Focused Tests
- Run `pytest tests/skills/test_setup_contract.py` after setup-skill changes. The current executable test covers active-interpreter path preservation; review the remaining approval and reporting instructions statically.
- Run `pytest tests/skills/test_ascend_npu_collect_contract.py` after
collection-skill changes. The v1 evaluation harness accepts only
`diagnostic-readonly` suites, so collection coverage stays in the static
collection-skill changes. Collection workflows are not `diagnostic-readonly`
or `agent-cli` eval suites, so collection coverage stays in the static
contract.
- Run `pytest tests/skills` after diagnostic-skill changes. Static contracts
check current surfaces and safety boundaries; suite/case evaluations cover
Expand Down
5 changes: 3 additions & 2 deletions tests/skills/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,17 +12,18 @@ tool-call traces, deterministic grading, and runner adapters.
| --- | --- |
| `schemas/` | Human-readable v1 suite and case descriptor contracts. |
| `harness/` | Strict descriptor loading, synthetic fixture construction, trace grading, runner protocols, and local artifacts. |
| `suites/` | Reviewed skill-specific objectives, decision branches, cases, and declarative SQLite fixtures. |
| `suites/` | Reviewed skill-specific and Agent CLI e2e objectives, decision branches, cases, and declarative SQLite fixtures. |
| `README.md` | Descriptor, gateway, run-record, grading, and local command reference. |
| `test_*_contract.py` | Static and executable contracts for shipped `SKILL.md` files. |

## Invariants
- Keep evaluation definitions declarative; do not embed shell, Python, Jinja, or arbitrary expressions in YAML.
- Diagnostic suites use generated SnapshotDB files only. Never expose, import, or deserialize pickle fixtures in an agent run.
- Diagnostic suites use generated SnapshotDB files only. Never expose, import, or deserialize pickle fixtures in an agent run. The `agent-cli` e2e suite may require semantic `pt_snap.import`; still do not materialize pickle inputs.
- Treat forbidden operations as hard failures and grade allowed tool calls by normalized semantics rather than shell text.
- Keep live model execution local and explicit. Normal pytest coverage must not require network access, provider credentials, or a model runtime.
- Write generated transcripts, databases, scores, and reports only under temporary directories or the ignored `.skill-evals/` directory.

## Focused Tests
- Run `pytest tests/skills` for harness, descriptor, fixture, grader, or skill contract changes.
- Run `python -m tests.skills validate tests/skills/suites/<skill>/suite.yaml` after editing a suite or case descriptor.
- Run `python -m tests.skills baseline tests/skills/suites/pt-snap-agent-e2e/suite.yaml <runs-dir>` to emit task success rate, call count, output size, and error-conclusion metrics.
38 changes: 31 additions & 7 deletions tests/skills/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,17 +10,20 @@ Each suite has one `suite.yaml` and explicit case references under
`suites/<skill>/`. The versioned contracts are documented in `schemas/` and
enforced by `harness/descriptors.py` with unknown fields rejected.

- A suite identifies the `SKILL.md`, result classifications, scored objectives,
required decision branches, sandbox defaults, semantic tool policy, and cases.
Runner defaults (`runner`, `repetitions`, `timeout_seconds`, `sandbox`) are
validated and exposed as typed values on the loaded suite so adapters can
enforce them.
- A suite identifies the skill or evaluation contract, result classifications,
scored objectives, required decision branches, sandbox defaults, semantic tool
policy, and cases. Supported profiles are `diagnostic-readonly` and
`agent-cli`. Runner defaults (`runner`, `repetitions`, `timeout_seconds`,
`sandbox`) are validated and exposed as typed values on the loaded suite so
adapters can enforce them.
- A case identifies its prompt, optional synthetic SnapshotDB, covered branches,
required tool actions, partial ordering, forbidden actions, oracle facts,
classification bounds, unknowns, and claim-to-tool evidence links. Actions may
declare `expect_output`, a mapping that must appear in the matched call's
recorded output before the action counts as matched. Cases may cap the total
call count with `expected_tools.max_calls`; zero enforces refusal-only cases.
recorded output before the action counts as matched. Actions may also declare
`status` (`success` or `error`; default `success`) so recovery cases can
require structured failures. Cases may cap the total call count with
`expected_tools.max_calls`; zero enforces refusal-only cases.
- Descriptor paths are repository-relative and cannot contain `..`. Fixture
mount paths must be normalized absolute POSIX paths under `/fixtures/`.
- Diagnostic fixtures are declarative SQLite databases. Pickle inputs are never
Expand All @@ -30,6 +33,7 @@ Validate a suite locally:

```bash
python -m tests.skills validate tests/skills/suites/pt-snap-memory-leak/suite.yaml
python -m tests.skills validate tests/skills/suites/pt-snap-agent-e2e/suite.yaml
```

## Tool Gateway
Expand Down Expand Up @@ -87,5 +91,25 @@ call IDs. Runs that omit required result fields or submit malformed result
shapes also hard-fail. Formatting and prose style receive no deterministic
score.

## Agent CLI baseline (issue #136)

`tests/skills/suites/pt-snap-agent-e2e/` grades the parent issue's seven
end-to-end scenarios. It is an `agent-cli` evaluation contract, not a shipped
skill. Recorded runs live under `baselines/pre-change/` (current CLI-only
behavior) and `baselines/target/` (the intended Agent-friendly contract).

Comparison metrics are task success rate, mean call count, mean output bytes,
and error-conclusion rate:

```bash
python -m tests.skills baseline \
tests/skills/suites/pt-snap-agent-e2e/suite.yaml \
tests/skills/suites/pt-snap-agent-e2e/baselines/pre-change \
--output .skill-evals/baselines/pre-change.json
```

Diagnostic fixtures remain declarative SQLite databases. The import scenario
records `pt_snap.import` semantically and never materializes a pickle.

Generated transcripts and reports belong under `.skill-evals/`, which is
ignored by Git. Normal `pytest` runs never invoke a live model.
36 changes: 36 additions & 0 deletions tests/skills/__main__.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
from .harness.artifacts import write_run_artifacts
from .harness.descriptors import load_suite
from .harness.grader import RunRecord, grade_run
from .harness.metrics import collect_case_metrics, summarize_metrics


def _validate(args: argparse.Namespace) -> int:
Expand Down Expand Up @@ -38,6 +39,29 @@ def _grade(args: argparse.Namespace) -> int:
return 0 if grade.passed else 1


def _baseline(args: argparse.Namespace) -> int:
suite = load_suite(args.suite)
rows = []
missing = []
for case in suite.cases:
run_path = args.runs / f"{case.id}.json"
if not run_path.is_file():
missing.append(str(run_path))
continue
run = RunRecord.from_mapping(json.loads(run_path.read_text()))
grade = grade_run(suite, case, run)
rows.append(collect_case_metrics(case, run, grade))
if missing:
raise SystemExit("missing run records:\n" + "\n".join(missing))
summary = summarize_metrics(suite.id, rows)
payload = json.dumps(summary.to_mapping(), indent=2, sort_keys=True)
if args.output is not None:
args.output.parent.mkdir(parents=True, exist_ok=True)
args.output.write_text(payload + "\n")
Comment thread
codeasier marked this conversation as resolved.
Resolved
print(payload)
return 0


def main() -> int:
parser = argparse.ArgumentParser(description="Validate and grade local skill evaluations")
subparsers = parser.add_subparsers(dest="command", required=True)
Expand All @@ -53,6 +77,18 @@ def main() -> int:
grade_parser.add_argument("--output", type=Path)
grade_parser.set_defaults(handler=_grade)

baseline_parser = subparsers.add_parser(
"baseline", help="score a directory of recorded runs and emit comparison metrics"
)
baseline_parser.add_argument("suite", type=Path)
baseline_parser.add_argument(
"runs",
type=Path,
help="directory containing <case-id>.json run records",
)
baseline_parser.add_argument("--output", type=Path)
baseline_parser.set_defaults(handler=_baseline)

args = parser.parse_args()
return args.handler(args)

Expand Down
5 changes: 5 additions & 0 deletions tests/skills/harness/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,11 @@
from .descriptors import DescriptorError, EvalCase, EvalSuite, SandboxPolicy, load_suite
from .gateway import RecordingToolGateway, ToolDeniedError
from .grader import GradeResult, RunRecord, ToolCall, grade_run
from .metrics import BaselineSummary, CaseMetrics, collect_case_metrics, summarize_metrics

__all__ = [
"BaselineSummary",
"CaseMetrics",
"DescriptorError",
"EvalCase",
"EvalSuite",
Expand All @@ -14,6 +17,8 @@
"SandboxPolicy",
"ToolCall",
"ToolDeniedError",
"collect_case_metrics",
"grade_run",
"load_suite",
"summarize_metrics",
]
20 changes: 16 additions & 4 deletions tests/skills/harness/descriptors.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@
import yaml

REPO_ROOT = Path(__file__).resolve().parents[3]
SUPPORTED_PROFILES = frozenset({"diagnostic-readonly", "agent-cli"})
SUPPORTED_ACTION_STATUSES = frozenset({"success", "error"})


class DescriptorError(ValueError):
Expand All @@ -28,6 +30,7 @@ class ExpectedAction:
operation: str
match: dict[str, Any]
expect_output: dict[str, Any] = field(default_factory=dict)
status: str = "success"


@dataclass(frozen=True)
Expand Down Expand Up @@ -241,18 +244,25 @@ def _load_case(
raw_action,
context=f"{path}: action {index}",
required={"id", "operation", "match"},
allowed={"id", "operation", "match", "expect_output"},
allowed={"id", "operation", "match", "expect_output", "status"},
)
if not isinstance(action["match"], dict):
raise DescriptorError(f"{path}: action {index}.match must be a mapping")
if "expect_output" in action and not isinstance(action["expect_output"], dict):
raise DescriptorError(f"{path}: action {index}.expect_output must be a mapping")
status = _string(action.get("status", "success"), f"{path}: action {index}.status")
if status not in SUPPORTED_ACTION_STATUSES:
raise DescriptorError(
f"{path}: action {index}.status must be one of "
f"{', '.join(sorted(SUPPORTED_ACTION_STATUSES))}"
)
actions.append(
ExpectedAction(
id=_string(action["id"], f"{path}: action id"),
operation=_string(action["operation"], f"{path}: action operation"),
match=dict(action["match"]),
expect_output=dict(action.get("expect_output", {})),
status=status,
)
)
action_ids = {action.id for action in actions}
Expand Down Expand Up @@ -473,8 +483,10 @@ def load_suite(path: Path, *, repo_root: Path = REPO_ROOT) -> EvalSuite:
if f"name: {skill_name}" not in skill_path.read_text():
raise DescriptorError(f"{path}: expected skill name is not present in {skill_path}")
profile = _string(skill["profile"], f"{path}: skill.profile")
if profile != "diagnostic-readonly":
raise DescriptorError(f"{path}: v1 supports only diagnostic-readonly suites")
if profile not in SUPPORTED_PROFILES:
raise DescriptorError(
f"{path}: v1 supports only {', '.join(sorted(SUPPORTED_PROFILES))} suites"
)

result_contract = _strict_mapping(
data["result_contract"],
Expand Down Expand Up @@ -568,7 +580,7 @@ def load_suite(path: Path, *, repo_root: Path = REPO_ROOT) -> EvalSuite:
}
for key, expected in expected_sandbox.items():
if sandbox[key] != expected:
raise DescriptorError(f"{path}: diagnostic sandbox requires {key}: {expected}")
raise DescriptorError(f"{path}: evaluation sandbox requires {key}: {expected}")
sandbox_policy = SandboxPolicy(
network=sandbox["network"],
home=sandbox["home"],
Expand Down
2 changes: 1 addition & 1 deletion tests/skills/harness/grader.py
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,7 @@ def _match_actions(case: EvalCase, run: RunRecord) -> tuple[dict[str, str], dict
if call.id in used_calls:
continue
if (
call.status == "success"
call.status == action.status
and call.operation == action.operation
and _is_subset(action.match, call.arguments)
and _output_contains(action.expect_output, call.output)
Expand Down
113 changes: 113 additions & 0 deletions tests/skills/harness/metrics.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
from __future__ import annotations

import json
from collections.abc import Iterable
from dataclasses import dataclass
from typing import Any

from .descriptors import EvalCase
from .grader import GradeResult, RunRecord


def _normalize_text(value: str) -> str:
return " ".join(value.casefold().split())


def run_output_bytes(run: RunRecord) -> int:
"""Count UTF-8 bytes of recorded tool outputs, errors, and the final response."""
chunks: list[str] = [run.final_response]
for call in run.tool_calls:
if call.output is not None:
chunks.append(
json.dumps(call.output, ensure_ascii=False, sort_keys=True, separators=(",", ":"))
)
if call.error:
chunks.append(call.error)
return sum(len(chunk.encode("utf-8")) for chunk in chunks)


def is_error_conclusion(case: EvalCase, run: RunRecord) -> bool:
"""True when the run asserts a prohibited classification or prohibited claim."""
classification = run.result.get("classification")
if classification in case.prohibited_classifications:
return True
raw_claims = run.result.get("claims", [])
claim_text = " ".join(
str(claim.get("statement", "")) for claim in raw_claims if isinstance(claim, dict)
)
searchable = _normalize_text(f"{claim_text} {run.final_response}")
return any(_normalize_text(prohibited) in searchable for prohibited in case.prohibited_claims)


@dataclass(frozen=True)
class CaseMetrics:
case_id: str
passed: bool
score: float
call_count: int
output_bytes: int
error_conclusion: bool

def to_mapping(self) -> dict[str, Any]:
return {
"case_id": self.case_id,
"passed": self.passed,
"score": self.score,
"call_count": self.call_count,
"output_bytes": self.output_bytes,
"error_conclusion": self.error_conclusion,
}


@dataclass(frozen=True)
class BaselineSummary:
suite_id: str
case_count: int
passed_count: int
task_success_rate: float
mean_call_count: float
mean_output_bytes: float
error_conclusion_rate: float
cases: tuple[CaseMetrics, ...]

def to_mapping(self) -> dict[str, Any]:
return {
"suite_id": self.suite_id,
"case_count": self.case_count,
"passed_count": self.passed_count,
"task_success_rate": self.task_success_rate,
"mean_call_count": self.mean_call_count,
"mean_output_bytes": self.mean_output_bytes,
"error_conclusion_rate": self.error_conclusion_rate,
"cases": [case.to_mapping() for case in self.cases],
}


def collect_case_metrics(case: EvalCase, run: RunRecord, grade: GradeResult) -> CaseMetrics:
return CaseMetrics(
case_id=case.id,
passed=grade.passed,
score=grade.score,
call_count=len(run.tool_calls),
output_bytes=run_output_bytes(run),
error_conclusion=is_error_conclusion(case, run),
)


def summarize_metrics(suite_id: str, rows: Iterable[CaseMetrics]) -> BaselineSummary:
cases = tuple(rows)
case_count = len(cases)
if case_count == 0:
raise ValueError("baseline summary requires at least one case")
passed_count = sum(case.passed for case in cases)
error_count = sum(case.error_conclusion for case in cases)
return BaselineSummary(
suite_id=suite_id,
case_count=case_count,
passed_count=passed_count,
task_success_rate=round(passed_count / case_count, 4),
mean_call_count=round(sum(case.call_count for case in cases) / case_count, 4),
mean_output_bytes=round(sum(case.output_bytes for case in cases) / case_count, 4),
error_conclusion_rate=round(error_count / case_count, 4),
cases=cases,
)
4 changes: 3 additions & 1 deletion tests/skills/schemas/case-v1.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,9 @@ fields:
optional: [max_calls]
max_calls_minimum: 0
action_required: [id, operation, match]
action_optional: [expect_output]
action_optional: [expect_output, status]
action_status_values: [success, error]
action_status_default: success
order_required: [before, after]
forbidden_required: [operation]
oracle:
Expand Down
2 changes: 1 addition & 1 deletion tests/skills/schemas/suite-v1.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ fields:
required: [id, title, description]
skill:
required: [path, expected_name, profile]
profile_values: [diagnostic-readonly]
profile_values: [diagnostic-readonly, agent-cli]
result_contract:
required: [required_fields, classifications]
required_result_fields: [classification, facts, claims, unknowns]
Expand Down
Loading
Loading