Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 78 additions & 0 deletions tests/skills/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,84 @@ call IDs. Runs that omit required result fields or submit malformed result
shapes also hard-fail. Formatting and prose style receive no deterministic
score.

## Final-answer minimum validity (issue #186)

Suite and case descriptors may opt in with a top-level `final_answer` block:

```yaml
final_answer:
require_nonempty: true
required_conclusions:
database: [/fixtures/target.db]
device: [device 1, device_id=1]
```

`require_nonempty` is a boolean, defaulting to `true` when the block is present.
`required_conclusions` defaults to `{}` and maps conclusion names to non-empty
lists of accepted non-empty phrases. Each named conclusion needs at least one
phrase in **`final_response` itself**, after Unicode case folding and whitespace
collapsing. This is literal substring matching, not a semantic judge: it does not
validate arbitrary paraphrases, negation, factual truth, or prose style. Authors
should declare meaningful phrases and accepted language variants; a non-empty
check alone deliberately accepts unrelated text.

- A case block **replaces the entire suite block**, rather than merging it.
An omitted case block inherits the suite policy. With both omitted, legacy v1
grading remains unchanged and the answer check reports `checked: false`.
- `{require_nonempty: false}` with no required conclusions explicitly disables
the check for a case. `{}` enables just the default non-empty check. Required
conclusions still apply if `require_nonempty` is false.
- Enabled checks apply to every classification, including correct refusals,
blocked and unavailable results. Facts/claims in `result` cannot substitute for
conclusions that the user must see in the final answer.
- Omitted `final_response` still defaults to `""` for legacy records. An explicit
null, boolean, number, list or mapping is rejected by the record loader; none
is converted to text. Unknown policy fields and malformed policy types are
rejected by the descriptor loader.

Answer failure is a hard gate on overall `passed`. The existing `score` remains
the trace/result objective score (so it may be 100 even when `passed` is false).
Grade JSON, `score.json` artifacts and each baseline case separately expose
`trace_result` (`passed`, `score`) and `final_answer` (`checked`, nullable `passed`,
`violations`). An unchecked answer is not evidence of answer quality.
`execution_evidence.runner_execution_verified: false` explicitly records that
grading cannot verify runner execution. `grade` only grades a supplied record;
`baseline` only rescores recorded baselines. Suite runner defaults, recorded tool
calls and a perfect grade are not proof of a fresh agent/model run. Any actual
runner execution evidence must be collected separately by the adapter.

The Agent CLI suite opts in for all ten cases with named minimum conclusions;
its target records remain unchanged. Deterministic mutation tests reject empty,
whitespace-only and unrelated answers without live models or cloud judges.

### 最终答复最低有效性(中文)

suite 和 case 可用上述顶层 `final_answer` 声明确定性检查。
声明块存在时,布尔字段 `require_nonempty` 默认 `true`;
`required_conclusions` 默认 `{}`,将结论名称映射到非空候选短语列表。
每项结论都必须在 **`final_response` 本身**匹配至少一个短语;匹配先进行
Unicode 大小写折叠并合并空白,再检查字面子串。这不判断任意同义改写、否定关系、
事实真伪或文风;作者应声明有意义的短语和可接受的语言变体。仅非空检查仍允许无关文本。

- case 声明**整块替换** suite 策略,不逐字段合并;case 省略则继承 suite。
两处均省略时保持旧版 v1 评分行为,报告 `checked: false`。
- `{require_nonempty: false}` 且无必需结论时显式关闭检查;`{}` 启用默认非空检查。
即使 `require_nonempty: false`,已声明的必需结论仍会检查。
- 已启用的检查适用于所有分类,包括正确拒绝、blocked 和 unavailable;
`result` 的 facts/claims 不能代替用户应在最终答复看到的结论。
- 旧记录省略 `final_response` 仍默认为空串;显式 null、布尔、数字、列表或映射
会被加载器拒绝,不会转成字符串。策略未知字段和类型错误也会被拒绝。

答复检查失败会使整体 `passed` 为 false;原有 `score` 仍是 trace/result 目标分数,
因此可能同时出现 100 分和整体不通过。grade JSON、`score.json` 和 baseline 各案例
分别报告 `trace_result`(通过状态和分数)与 `final_answer`(是否检查、可为空的通过
状态和违规原因)。未检查不代表答复合格。
`execution_evidence.runner_execution_verified: false` 表明评分无法验证真实 runner
执行:`grade` 只评分输入记录,`baseline` 只重评历史记录。suite runner 配置、记录的
工具调用和满分都不能证明新执行了 agent/model;真实执行证据需由 adapter 另行采集。
Agent CLI 的全部十个案例已启用命名结论检查,target 记录未改写;确定性变异测试覆盖
空串、纯空白和无关答复,无需实时模型或云端 judge。

## Agent CLI baseline (issue #136)

`tests/skills/suites/pt-snap-agent-e2e/` grades the issue #136 end-to-end
Expand Down
43 changes: 43 additions & 0 deletions tests/skills/harness/descriptors.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,12 @@ class SandboxPolicy:
clear_environment: tuple[str, ...]


@dataclass(frozen=True)
class FinalAnswerPolicy:
require_nonempty: bool = True
required_conclusions: tuple[tuple[str, tuple[str, ...]], ...] = ()


@dataclass(frozen=True)
class EvalCase:
id: str
Expand All @@ -63,6 +69,7 @@ class EvalCase:
max_tool_calls: int | None = None
additional_fixtures: tuple[dict[str, Any], ...] = ()
writable_outputs: tuple[str, ...] = ()
final_answer: FinalAnswerPolicy | None = None


@dataclass(frozen=True)
Expand All @@ -86,6 +93,7 @@ class EvalSuite:
timeout_seconds: int
sandbox: SandboxPolicy
required_result_fields: tuple[str, ...]
final_answer: FinalAnswerPolicy | None = None

@property
def objectives_by_id(self) -> dict[str, Objective]:
Expand Down Expand Up @@ -148,6 +156,29 @@ def _positive_int(value: Any, context: str, *, allow_zero: bool = False) -> int:
return value


def _final_answer_policy(value: Any, context: str) -> FinalAnswerPolicy:
policy = _strict_mapping(
value,
context=context,
required=set(),
allowed={"require_nonempty", "required_conclusions"},
)
require_nonempty = policy.get("require_nonempty", True)
if not isinstance(require_nonempty, bool):
raise DescriptorError(f"{context}.require_nonempty must be a boolean")
raw_conclusions = policy.get("required_conclusions", {})
if not isinstance(raw_conclusions, dict):
raise DescriptorError(f"{context}.required_conclusions must be a mapping")
conclusions = []
for name, raw_alternatives in raw_conclusions.items():
name = _string(name, f"{context}: conclusion name")
alternatives = _string_list(raw_alternatives, f"{context}: conclusion {name}")
if not alternatives:
raise DescriptorError(f"{context}: conclusion {name} requires at least one phrase")
conclusions.append((name, alternatives))
return FinalAnswerPolicy(require_nonempty, tuple(conclusions))


def _safe_path(base: Path, raw_path: Any, context: str) -> Path:
text = _string(raw_path, context)
relative = Path(text)
Expand Down Expand Up @@ -252,6 +283,7 @@ def _load_case(
"additional_fixtures",
"writable_outputs",
"dialog",
"final_answer",
"expected_tools",
"oracle",
"scoring",
Expand Down Expand Up @@ -479,6 +511,11 @@ def _load_case(
max_tool_calls=max_tool_calls,
additional_fixtures=tuple(additional_fixtures),
writable_outputs=writable_outputs,
final_answer=(
_final_answer_policy(data["final_answer"], f"{path}: final_answer")
if "final_answer" in data
else None
),
)


Expand All @@ -504,6 +541,7 @@ def load_suite(path: Path, *, repo_root: Path = REPO_ROOT) -> EvalSuite:
"metadata",
"skill",
"result_contract",
"final_answer",
"objectives",
"coverage",
"defaults",
Expand Down Expand Up @@ -728,4 +766,9 @@ def load_suite(path: Path, *, repo_root: Path = REPO_ROOT) -> EvalSuite:
timeout_seconds=timeout_seconds,
sandbox=sandbox_policy,
required_result_fields=required_result_fields,
final_answer=(
_final_answer_policy(data["final_answer"], f"{path}: final_answer")
if "final_answer" in data
else None
),
)
43 changes: 41 additions & 2 deletions tests/skills/harness/grader.py
Original file line number Diff line number Diff line change
Expand Up @@ -39,10 +39,13 @@ def from_mapping(cls, data: dict[str, Any]) -> RunRecord:
calls = data.get("tool_calls", [])
if not isinstance(calls, list) or not isinstance(data.get("result"), dict):
raise ValueError("run record requires tool_calls list and result mapping")
final_response = data.get("final_response", "")
if not isinstance(final_response, str):
raise ValueError("run record final_response must be a string")
return cls(
tool_calls=tuple(ToolCall.from_mapping(call) for call in calls),
result=dict(data["result"]),
final_response=str(data.get("final_response", "")),
final_response=final_response,
)


Expand All @@ -63,11 +66,24 @@ class GradeResult:
objectives: tuple[ObjectiveGrade, ...]
matched_actions: dict[str, str]
contract_violations: tuple[str, ...] = ()
trace_result_passed: bool = False
final_answer_passed: bool | None = None
final_answer_violations: tuple[str, ...] = ()

def to_mapping(self) -> dict[str, Any]:
return {
"passed": self.passed,
"score": self.score,
"trace_result": {"passed": self.trace_result_passed, "score": self.score},
"final_answer": {
"checked": self.final_answer_passed is not None,
"passed": self.final_answer_passed,
"violations": list(self.final_answer_violations),
},
"execution_evidence": {
"kind": "record_grading",
"runner_execution_verified": False,
},
"gate_violations": list(self.gate_violations),
"contract_violations": list(self.contract_violations),
"matched_actions": self.matched_actions,
Expand Down Expand Up @@ -299,10 +315,29 @@ def _result_contract_checks(suite: EvalSuite, run: RunRecord) -> tuple[str, ...]
return tuple(violations)


def _final_answer_checks(
suite: EvalSuite, case: EvalCase, run: RunRecord
) -> tuple[bool | None, tuple[str, ...]]:
policy = case.final_answer if case.final_answer is not None else suite.final_answer
if policy is None or (not policy.require_nonempty and not policy.required_conclusions):
return None, ()
if not isinstance(run.final_response, str):
return False, ("final_response must be a string",)
text = _normalize_text(run.final_response)
violations = []
if policy.require_nonempty and not text:
violations.append("final_response must be non-empty")
for name, alternatives in policy.required_conclusions:
if not any(_normalize_text(phrase) in text for phrase in alternatives):
violations.append(f"final_response is missing required conclusion: {name}")
return not violations, tuple(violations)


def grade_run(suite: EvalSuite, case: EvalCase, run: RunRecord) -> GradeResult:
matched, positions = _match_actions(case, run)
gate_violations = _safety_checks(suite, case, run)
contract_violations = _result_contract_checks(suite, run)
final_answer_passed, final_answer_violations = _final_answer_checks(suite, case, run)
scorer_results = {
"safety": (not gate_violations, gate_violations),
"tool_path": _tool_path_score(suite, case, run, matched, positions),
Expand Down Expand Up @@ -340,11 +375,15 @@ def grade_run(suite: EvalSuite, case: EvalCase, run: RunRecord) -> GradeResult:

by_id = {grade.objective_id: grade for grade in grades}
mandatory_passed = all(by_id[objective_id].passed for objective_id in case.mandatory_objectives)
trace_result_passed = not gate_violations and not contract_violations and mandatory_passed
return GradeResult(
passed=not gate_violations and not contract_violations and mandatory_passed,
passed=trace_result_passed and final_answer_passed is not False,
score=round(sum(grade.earned for grade in grades), 2),
gate_violations=gate_violations,
objectives=tuple(grades),
matched_actions=matched,
contract_violations=contract_violations,
trace_result_passed=trace_result_passed,
final_answer_passed=final_answer_passed,
final_answer_violations=final_answer_violations,
)
16 changes: 16 additions & 0 deletions tests/skills/harness/metrics.py
Original file line number Diff line number Diff line change
Expand Up @@ -47,12 +47,21 @@ class CaseMetrics:
call_count: int
output_bytes: int
error_conclusion: bool
trace_result_passed: bool
final_answer_passed: bool | None
final_answer_violations: tuple[str, ...]

def to_mapping(self) -> dict[str, Any]:
return {
"case_id": self.case_id,
"passed": self.passed,
"score": self.score,
"trace_result": {"passed": self.trace_result_passed, "score": self.score},
"final_answer": {
"checked": self.final_answer_passed is not None,
"passed": self.final_answer_passed,
"violations": list(self.final_answer_violations),
},
"call_count": self.call_count,
"output_bytes": self.output_bytes,
"error_conclusion": self.error_conclusion,
Expand All @@ -73,6 +82,10 @@ class BaselineSummary:
def to_mapping(self) -> dict[str, Any]:
return {
"suite_id": self.suite_id,
"execution_evidence": {
"kind": "recorded_baseline_rescore",
"runner_execution_verified": False,
},
"case_count": self.case_count,
"passed_count": self.passed_count,
"task_success_rate": self.task_success_rate,
Expand All @@ -91,6 +104,9 @@ def collect_case_metrics(case: EvalCase, run: RunRecord, grade: GradeResult) ->
call_count=len(run.tool_calls),
output_bytes=run_output_bytes(run),
error_conclusion=is_error_conclusion(case, run),
trace_result_passed=grade.trace_result_passed,
final_answer_passed=grade.final_answer_passed,
final_answer_violations=grade.final_answer_violations,
)


Expand Down
9 changes: 9 additions & 0 deletions tests/skills/schemas/case-v1.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,15 @@ required:
fields:
task:
required: [initial_user]
final_answer:
optional: [require_nonempty, required_conclusions]
require_nonempty_type: boolean
require_nonempty_default: true
required_conclusions_type: mapping-of-names-to-nonempty-lists-of-nonempty-phrases
matching: casefold-and-collapse-whitespace-substring-any-phrase-per-conclusion
rule: replaces-entire-suite-final-answer-policy-for-all-classifications
omitted: inherit-suite-policy
disabled: require_nonempty-false-and-no-required-conclusions
fixture:
required: [builder, definition, mount_path, read_only]
builder_values: [synthetic-snapshotdb]
Expand Down
8 changes: 8 additions & 0 deletions tests/skills/schemas/suite-v1.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,14 @@ fields:
result_contract:
required: [required_fields, classifications]
required_result_fields: [classification, facts, claims, unknowns]
final_answer:
optional: [require_nonempty, required_conclusions]
require_nonempty_type: boolean
require_nonempty_default: true
required_conclusions_type: mapping-of-names-to-nonempty-lists-of-nonempty-phrases
matching: casefold-and-collapse-whitespace-substring-any-phrase-per-conclusion
rule: optional-hard-gate-for-all-classifications-case-block-replaces-suite-block
omitted: unchecked-for-legacy-compatibility
objectives:
item_required: [id, mode, scorer]
mode_values: [gate, score]
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,8 @@
schema_version: 1
final_answer:
required_conclusions:
helper: [pt-snap-helper]
host_reload: [restart the agent, restart your agent]
kind: skill-eval-case
id: cli-only-help-discovery
title: CLI-only agent finds helper guidance from help and skill list
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,8 @@
schema_version: 1
final_answer:
required_conclusions:
database: [/fixtures/target.db]
device: [device 1]
kind: skill-eval-case
id: explicit-multi-target
title: Agent uses the user-specified database and device, not the first discovered ones
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,8 @@
schema_version: 1
final_answer:
required_conclusions:
database: [/outputs/imported.db]
focus: [focus was updated, focus updated]
kind: skill-eval-case
id: import-success
title: Successful import reports the published database and focus change
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,8 @@
schema_version: 1
final_answer:
required_conclusions:
candidate: [4096-byte candidate, candidate of 4096 bytes]
uncertainty: [not proof, unconfirmed]
kind: skill-eval-case
id: live-blocks-unconfirmed
title: End-of-trace live blocks are not a confirmed leak
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,8 @@
schema_version: 1
final_answer:
required_conclusions:
unavailable: [database is missing, database not found]
failed_lookup: [failed lookup]
kind: skill-eval-case
id: missing-database
title: A missing database is a failure, not an empty successful result
Expand Down
4 changes: 4 additions & 0 deletions tests/skills/suites/pt-snap-agent-e2e/cases/no-device.yaml
Original file line number Diff line number Diff line change
@@ -1,4 +1,8 @@
schema_version: 1
final_answer:
required_conclusions:
unavailable: [no discoverable device, no device]
failed_lookup: [failed lookup]
kind: skill-eval-case
id: no-device
title: A database without discoverable devices is a failure, not an empty result
Expand Down
Loading
Loading