Repository navigation
feat: add local skill evaluation framework - #110
Conversation
| if raw_fixture["read_only"] is not True: | ||
| raise DescriptorError(f"{path}: diagnostic fixtures must be read-only") | ||
| mount_path = _string(raw_fixture["mount_path"], f"{path}: fixture.mount_path") | ||
| if not mount_path.startswith("/fixtures/"): |
There was a problem hiding this comment.
【中】mount_path 仅做字符串前缀检查:/fixtures/../project/result.db 能通过 startswith("/fixtures/") 校验,但归一化后落在 /fixtures 之外,可突破声明的只读 fixture 边界。建议按 POSIX 路径解析,拒绝 .. 与空目标,并校验归一化后仍位于 /fixtures/ 之下。
There was a problem hiding this comment.
已在 94341b8 修复:fixture.mount_path 现在要求绝对 POSIX 路径、拒绝包含 .. 的路径、要求 posixpath.normpath 归一化形式(拒绝 //、/./、尾部斜杠等),最后校验归一化结果仍位于 /fixtures/ 之下。/fixtures/../project/result.db 现在会在 cannot contain '..' 处被拒绝。回归测试:test_case_rejects_mount_path_traversal、test_case_rejects_unnormalized_mount_path。
| ) | ||
| required_branches = _string_list(coverage["required_branches"], f"{path}: required_branches") | ||
|
|
||
| defaults = _strict_mapping( |
There was a problem hiding this comment.
【中】defaults.runner/repetitions/timeout_seconds/sandbox(clear_environment 等) 在校验后被丢弃,未写入 EvalSuite(descriptors.py:602),RunRequest(runners.py:11)也无法携带它们。因此描述文件里声明的安全隔离、超时、重复次数、环境清理策略无法被 runner 适配器执行。建议把这些设置存入类型化模型并交由 runner 协调器使用。
There was a problem hiding this comment.
已在 94341b8 修复:EvalSuite 现以类型化字段保留 runner、repetitions、timeout_seconds、sandbox(新增 SandboxPolicy,含 clear_environment)以及 required_result_fields;RunRequest 携带整个 suite,runner 适配器可直接读取并执行这些声明。进程级隔离的实际执行归适配器负责(本 PR 无 live runner,ScriptedRunner 仅用于确定性测试);README 与 schemas/suite-v1.yaml 已标注 exposed_on_loaded_suite: runner-adapters-enforce-these-values。测试:test_suite_exposes_runner_defaults_and_result_contract。
| matched: dict[str, str], | ||
| positions: dict[str, int], | ||
| ) -> tuple[float, tuple[str, ...]]: | ||
| checks: list[tuple[bool, str]] = [ |
There was a problem hiding this comment.
【中】tool-path 评分只要求必需动作与预算,不拒绝多余的允许列表内调用。pickle-refusal 用例声明 actions: [](pickle-refusal.yaml:12),代理即使对 pickle 路径调用 pt_snap.metadata/sqlite.readonly(甚至 status=error)仍可获得满分并通过。实测补充该调用后仍 passed=True、score=100。建议支持用例级调用限制/零调用预算,并让拒绝型用例的 tool-path 要求成为强制性的。
There was a problem hiding this comment.
已在 94341b8 修复:新增用例级 expected_tools.max_calls(最小 0,由 descriptors 校验),_tool_path_score 按用例强制执行。pickle-refusal 现声明 max_calls: 0,并将 tools.correct_path 加入 mandatory_objectives:任何工具调用(即使 status=error)都会以 case tool call budget exceeded 硬失败该用例;零调用运行仍满分。你复现的场景(对 pickle 路径补调用仍 passed=True/100)现在 passed=False。回归测试:test_refusal_case_hard_fails_any_tool_call、test_refusal_case_accepts_a_call_free_run。
| required={"required_fields", "classifications"}, | ||
| allowed={"required_fields", "classifications"}, | ||
| ) | ||
| required_result_fields = set( |
There was a problem hiding this comment.
【中】result_contract.required_fields 在校验后被丢弃,EvalSuite 没有保留该契约,RunRecord.from_mapping(grader.py:38)也只要求任意的 result mapping。实测 pickle-refusal 用例省略 claims 仍可通过(passed=True, score=100)。建议在 EvalSuite 中保留 required result fields,并在评分前校验其存在及形状。
There was a problem hiding this comment.
已在 94341b8 修复:required_result_fields 保留在 EvalSuite 上;grade_run 在评分前按 suite 校验 result 的字段存在性与形状(classification: str、facts: dict、claims: list、unknowns: list),违规写入新增的 GradeResult.contract_violations 并使 passed=False(与 safety gate 分离,避免混淆语义)。你实测的“pickle-refusal 省略 claims 仍 passed=True/100”现在失败。回归测试:test_grader_enforces_required_result_fields、test_grader_rejects_malformed_result_shapes。
| return _fraction(checks) | ||
|
|
||
|
|
||
| def _facts_score(case: EvalCase, run: RunRecord) -> tuple[float, tuple[str, ...]]: |
There was a problem hiding this comment.
【高】评分器信任提交的 facts,未与工具输出核对:_facts_score 直接读取 run.result.facts,而 _evidence_score(grader.py:184)仅校验 claims 引用的 call ID,从不检查 call.output。运行可以把工具输出全部留空、直接把 oracle 值抄进 result,仍能拿到 100 分。已用现有 passing fixture 复现:ToolCall 全部不带 output 时 grade_run 返回 passed=True、score=100。建议为每个 fact/claim 增加基于所匹配 call 实际输出的断言(output predicates/extractors)。
There was a problem hiding this comment.
已在 94341b8 修复:action 支持声明 expect_output(映射子集断言,输出为行列表时按行包含匹配),_match_actions 将其纳入匹配条件——输出不含断言的动作视为未匹配,连带 tool_path、偏序与 evidence 链接失败。三个 SnapshotDB 用例已把 oracle facts 锚定到真实模板输出列(与 test_fixtures.py 实测一致):inspect_peaks→{peak_reserved: 8192}、inspect_gaps→{reserved_active_gap_at_reserved_peak: 7168}、inspect_candidates→{size: 4096}、inspect_lifecycle→{id: 2, action: 5}、inspect_blocks→{allocEventId: 3}、inspect_events→{id: 3, action: 4}。你复现的“输出全部留空仍 passed=True/100”现在 passed=False。回归测试:test_grader_rejects_facts_without_grounded_tool_outputs。
Address PR #110 review feedback: - Harden fixture.mount_path validation with POSIX normalization so traversal outside /fixtures/ (e.g. /fixtures/../x) is rejected. - Expose runner, repetitions, timeout_seconds, sandbox policy, and required result fields as typed values on EvalSuite so runner adapters can enforce declared defaults. - Add case-level expected_tools.max_calls (zero allowed) and make the pickle-refusal case require a call-free tool path. - Enforce the suite result contract before scoring: missing required fields or malformed shapes hard-fail via GradeResult contract violations. - Add per-action expect_output assertions so required actions must be grounded in recorded tool outputs; anchor memory-leak case facts to real template output columns and document the contracts.
Resolve conflicts for the base switch from feat/skill-memory-leaks to main: - Keep main's pt-snap-memory-leak SKILL.md with its shell-quoting and untrusted-focus-value safety iteration. - Port main's expanded memory-leak contract assertions into the migrated tests/skills/test_memory_leak_contract.py and migrate the new fragmentation and peak-breakdown contract tests under tests/skills/ with the *_contract.py naming convention. - Combine AGENTS navigation: keep the tests/skills/ migration routing and main's installation-approval wording, scope rows, and invariants.
📝 Description
tests/skills/with versioned suite/case descriptors, strict validation, declared decision-branch coverage, and runner adapters.pt-snap-memory-leakscenarios for allocator/cache effects, pending frees, address reuse, and pickle refusal.🔗 Related Issue
Closes #109
Stacked on #104 only because the initial example suite evaluates the memory-leak skill added there. The framework is maintained on its own branch and worktree; after #104 merges, this PR can target
mainwithout changing its evaluation-framework diff.🧪 Testing
Commands run in the isolated evaluation-framework worktree:
PYTHONPATH=src pytest(712 passed, 1 xfailed)python -m tests.skills validate tests/skills/suites/pt-snap-memory-leak/suite.yamlruff check .black --check .python3 /Users/test1/.agents/skills/agents-docs/scripts/validate_agents.py . --strict(23 files, 0 errors, 0 warnings)git diff --check origin/feat/skill-memory-leaks...HEAD✅ Checklist
Snapshot Runtime Provenance
Snapshot provenance decision: no-update
Snapshot provenance no-update reason: This PR changes test-only skill evaluation infrastructure, synthetic SQLite definitions, and AGENTS navigation; no files under
src/pt_snap_cli/snapshot/**changed.📸 Screenshots (if applicable)
Not applicable.
📋 Additional Notes
Diagnostic suites never expose or deserialize pickle fixtures. Agent adapters must use an isolated
HOME, cleared focus environment, read-only project and fixture mounts, disabled network access, and the recording semantic tool gateway. Generated transcripts and scores remain under ignored.skill-evals/paths.