Skip to content

feat: add local skill evaluation framework - #110

Merged
codeasier merged 4 commits into
mainfrom
feat/skill-evaluation-framework
Aug 24, 2026
Merged

codeasier merged 4 commits into
mainfrom
feat/skill-evaluation-framework

Conversation

@codeasier

Copy link
Copy Markdown
Owner

📝 Description

  • Add a local-first skill evaluation framework under tests/skills/ with versioned suite/case descriptors, strict validation, declared decision-branch coverage, and runner adapters.
  • Grade normalized tool operations, partial ordering, budgets, oracle facts, classifications, unknowns, and claim-to-call evidence links; treat forbidden operations as hard failures.
  • Add a recording tool gateway, ignored local artifacts, and synthetic read-only SnapshotDB fixture construction without exposing pickle inputs.
  • Provide four initial pt-snap-memory-leak scenarios for allocator/cache effects, pending frees, address reuse, and pickle refusal.
  • Keep live model execution explicit and local; this PR adds no GitHub workflow or provider dependency.

🔗 Related Issue

Closes #109

Stacked on #104 only because the initial example suite evaluates the memory-leak skill added there. The framework is maintained on its own branch and worktree; after #104 merges, this PR can target main without changing its evaluation-framework diff.

🧪 Testing

  • Tests added/updated
  • Manual testing completed

Commands run in the isolated evaluation-framework worktree:

  • PYTHONPATH=src pytest (712 passed, 1 xfailed)
  • python -m tests.skills validate tests/skills/suites/pt-snap-memory-leak/suite.yaml
  • ruff check .
  • black --check .
  • python3 /Users/test1/.agents/skills/agents-docs/scripts/validate_agents.py . --strict (23 files, 0 errors, 0 warnings)
  • git diff --check origin/feat/skill-memory-leaks...HEAD

✅ Checklist

  • Code follows project guidelines
  • Documentation updated (if applicable)
  • No new warnings introduced
  • Changes tested locally

Snapshot Runtime Provenance

  • Snapshot provenance evaluation completed when snapshot runtime files change
    Snapshot provenance decision: no-update
    Snapshot provenance no-update reason: This PR changes test-only skill evaluation infrastructure, synthetic SQLite definitions, and AGENTS navigation; no files under src/pt_snap_cli/snapshot/** changed.

📸 Screenshots (if applicable)

Not applicable.

📋 Additional Notes

Diagnostic suites never expose or deserialize pickle fixtures. Agent adapters must use an isolated HOME, cleared focus environment, read-only project and fixture mounts, disabled network access, and the recording semantic tool gateway. Generated transcripts and scores remain under ignored .skill-evals/ paths.

if raw_fixture["read_only"] is not True:
raise DescriptorError(f"{path}: diagnostic fixtures must be read-only")
mount_path = _string(raw_fixture["mount_path"], f"{path}: fixture.mount_path")
if not mount_path.startswith("/fixtures/"):

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

【中】mount_path 仅做字符串前缀检查:/fixtures/../project/result.db 能通过 startswith("/fixtures/") 校验,但归一化后落在 /fixtures 之外,可突破声明的只读 fixture 边界。建议按 POSIX 路径解析,拒绝 .. 与空目标,并校验归一化后仍位于 /fixtures/ 之下。

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已在 94341b8 修复:fixture.mount_path 现在要求绝对 POSIX 路径、拒绝包含 .. 的路径、要求 posixpath.normpath 归一化形式(拒绝 //、/./、尾部斜杠等),最后校验归一化结果仍位于 /fixtures/ 之下。/fixtures/../project/result.db 现在会在 cannot contain '..' 处被拒绝。回归测试:test_case_rejects_mount_path_traversal、test_case_rejects_unnormalized_mount_path。

)
required_branches = _string_list(coverage["required_branches"], f"{path}: required_branches")

defaults = _strict_mapping(

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

【中】defaults.runner/repetitions/timeout_seconds/sandbox(clear_environment 等) 在校验后被丢弃,未写入 EvalSuite(descriptors.py:602),RunRequest(runners.py:11)也无法携带它们。因此描述文件里声明的安全隔离、超时、重复次数、环境清理策略无法被 runner 适配器执行。建议把这些设置存入类型化模型并交由 runner 协调器使用。

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已在 94341b8 修复:EvalSuite 现以类型化字段保留 runner、repetitions、timeout_seconds、sandbox(新增 SandboxPolicy,含 clear_environment)以及 required_result_fields;RunRequest 携带整个 suite,runner 适配器可直接读取并执行这些声明。进程级隔离的实际执行归适配器负责(本 PR 无 live runner,ScriptedRunner 仅用于确定性测试);README 与 schemas/suite-v1.yaml 已标注 exposed_on_loaded_suite: runner-adapters-enforce-these-values。测试:test_suite_exposes_runner_defaults_and_result_contract。

matched: dict[str, str],
positions: dict[str, int],
) -> tuple[float, tuple[str, ...]]:
checks: list[tuple[bool, str]] = [

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

【中】tool-path 评分只要求必需动作与预算,不拒绝多余的允许列表内调用。pickle-refusal 用例声明 actions: [](pickle-refusal.yaml:12),代理即使对 pickle 路径调用 pt_snap.metadata/sqlite.readonly(甚至 status=error)仍可获得满分并通过。实测补充该调用后仍 passed=True、score=100。建议支持用例级调用限制/零调用预算,并让拒绝型用例的 tool-path 要求成为强制性的。

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已在 94341b8 修复:新增用例级 expected_tools.max_calls(最小 0,由 descriptors 校验),_tool_path_score 按用例强制执行。pickle-refusal 现声明 max_calls: 0,并将 tools.correct_path 加入 mandatory_objectives:任何工具调用(即使 status=error)都会以 case tool call budget exceeded 硬失败该用例;零调用运行仍满分。你复现的场景(对 pickle 路径补调用仍 passed=True/100)现在 passed=False。回归测试:test_refusal_case_hard_fails_any_tool_call、test_refusal_case_accepts_a_call_free_run。

Comment thread tests/skills/harness/descriptors.py Outdated
required={"required_fields", "classifications"},
allowed={"required_fields", "classifications"},
)
required_result_fields = set(

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

【中】result_contract.required_fields 在校验后被丢弃,EvalSuite 没有保留该契约,RunRecord.from_mapping(grader.py:38)也只要求任意的 result mapping。实测 pickle-refusal 用例省略 claims 仍可通过(passed=True, score=100)。建议在 EvalSuite 中保留 required result fields,并在评分前校验其存在及形状。

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已在 94341b8 修复:required_result_fields 保留在 EvalSuite 上;grade_run 在评分前按 suite 校验 result 的字段存在性与形状(classification: str、facts: dict、claims: list、unknowns: list),违规写入新增的 GradeResult.contract_violations 并使 passed=False(与 safety gate 分离,避免混淆语义)。你实测的“pickle-refusal 省略 claims 仍 passed=True/100”现在失败。回归测试:test_grader_enforces_required_result_fields、test_grader_rejects_malformed_result_shapes。

return _fraction(checks)


def _facts_score(case: EvalCase, run: RunRecord) -> tuple[float, tuple[str, ...]]:

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

【高】评分器信任提交的 facts,未与工具输出核对:_facts_score 直接读取 run.result.facts,而 _evidence_score(grader.py:184)仅校验 claims 引用的 call ID,从不检查 call.output。运行可以把工具输出全部留空、直接把 oracle 值抄进 result,仍能拿到 100 分。已用现有 passing fixture 复现:ToolCall 全部不带 output 时 grade_run 返回 passed=True、score=100。建议为每个 fact/claim 增加基于所匹配 call 实际输出的断言(output predicates/extractors)。

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已在 94341b8 修复:action 支持声明 expect_output(映射子集断言,输出为行列表时按行包含匹配),_match_actions 将其纳入匹配条件——输出不含断言的动作视为未匹配,连带 tool_path、偏序与 evidence 链接失败。三个 SnapshotDB 用例已把 oracle facts 锚定到真实模板输出列(与 test_fixtures.py 实测一致):inspect_peaks→{peak_reserved: 8192}、inspect_gaps→{reserved_active_gap_at_reserved_peak: 7168}、inspect_candidates→{size: 4096}、inspect_lifecycle→{id: 2, action: 5}、inspect_blocks→{allocEventId: 3}、inspect_events→{id: 3, action: 4}。你复现的“输出全部留空仍 passed=True/100”现在 passed=False。回归测试:test_grader_rejects_facts_without_grounded_tool_outputs。

Address PR #110 review feedback:

- Harden fixture.mount_path validation with POSIX normalization so
  traversal outside /fixtures/ (e.g. /fixtures/../x) is rejected.
- Expose runner, repetitions, timeout_seconds, sandbox policy, and
  required result fields as typed values on EvalSuite so runner
  adapters can enforce declared defaults.
- Add case-level expected_tools.max_calls (zero allowed) and make the
  pickle-refusal case require a call-free tool path.
- Enforce the suite result contract before scoring: missing required
  fields or malformed shapes hard-fail via GradeResult contract
  violations.
- Add per-action expect_output assertions so required actions must be
  grounded in recorded tool outputs; anchor memory-leak case facts to
  real template output columns and document the contracts.
@codeasier
codeasier changed the base branch from feat/skill-memory-leaks to main August 24, 2026 15:06
Resolve conflicts for the base switch from feat/skill-memory-leaks to main:

- Keep main's pt-snap-memory-leak SKILL.md with its shell-quoting and
  untrusted-focus-value safety iteration.
- Port main's expanded memory-leak contract assertions into the migrated
  tests/skills/test_memory_leak_contract.py and migrate the new
  fragmentation and peak-breakdown contract tests under tests/skills/
  with the *_contract.py naming convention.
- Combine AGENTS navigation: keep the tests/skills/ migration routing
  and main's installation-approval wording, scope rows, and invariants.
@codeasier
codeasier merged commit 89f5ddd into main Aug 24, 2026
6 checks passed
@codeasier
codeasier deleted the feat/skill-evaluation-framework branch October 3, 2026 19:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE] Add a local skill evaluation framework

1 participant