Repository navigation
feat(tests): add Agent CLI end-to-end evaluation baseline - #145
Conversation
Freeze #136's pre-change Agent metrics so later JSON, helper, and pagination work can be compared against a recorded CLI-only score.
codeasier
left a comment
There was a problem hiding this comment.
交叉评审结论:Approve with low-severity follow-ups。
- 范围确认:仅测试 harness(
tests/skills/*),无产品代码改动;评分与指标数字已独立复算一致(target 7/7 满分 100;pre-change 0/7,error_conclusion_rate 0.8571,mean_call_count 1.0)。 - 建议合并前修复:#1(--output 父目录)、#2(基线行序)。
- 可跟踪为后续工作:#3(gateway 错误形状)、#4(信封契约)、#5(沙箱张力)、#6(测试缺口)。
评审溯源:3 位独立评审(qwen3.8-flash、deepseek-v4.1-flash、muse-spark-1.3-contributor,3/3 成功)+ 父会话逐条对照快照验证。
| "template": "leak_detection" | ||
| }, | ||
| "status": "error", | ||
| "output": { |
There was a problem hiding this comment.
Low:该错误形状无法由 RecordingToolGateway 产生。
status: "error" 且带结构化 output 是恢复/缺失目标用例的要求,但 harness/gateway.py:48-63 的异常路径只记 error 字符串(output 保持 None),从不同时记录两者。未来基于该 gateway 的 local-agent runner 会无法通过这些用例。建议明确约定或扩展 gateway 以支持结构化错误负载(wrong-template-recovery.json:11-16 同样如此)。
There was a problem hiding this comment.
Agreed; tracking as follow-up, no code in this PR. The target traces describe the #138 structured-error envelope. RecordingToolGateway still records exceptions as an error string with output=None, so a future local-agent runner cannot satisfy these cases until the gateway (or JSON error protocol) lands.
| | `pt_snap.help` | `{mentions_helper: true, helper_skill: "pt-snap-helper"}` when help points at the helper skill | | ||
| | `pt_snap.skill_list` | JSON listing; `json: true` in arguments | | ||
| | failed `pt_snap.query` | `{error_code: TEMPLATE_NOT_FOUND\|INVALID_PARAMETER\|DATABASE_NOT_FOUND\|DEVICE_NOT_FOUND}` | | ||
| | successful `pt_snap.query` | envelope with `db_path`, `device_id`, `returned`, `has_more`, and `rows` | |
There was a problem hiding this comment.
Low:信封契约声明与两个 target 基线矛盾。
此处称成功的 pt_snap.query 返回含 db_path、device_id、returned、has_more、rows 的信封;但 truncated-query.json:14-21 缺 db_path/device_id,live-blocks-unconfirmed.json:13-20 是裸列表。live-blocks 用例的 expect_output: {size: 4096} 目前仅靠 _output_contains 的列表分支(grader.py:96-97)通过;若是信封 dict 反而会失败(dict 匹配仅顶层,grader.py:94-95)。子任务 #137–#142 落地时二者必改其一。建议先在契约中限定适用范围,或把两个 target 基线归一化为完整信封。
There was a problem hiding this comment.
Agreed; tracking as follow-up, no code in this PR. The contract describes the intended success envelope, while truncated-query omits db_path/device_id and live-blocks-unconfirmed is a bare list (so {size: 4096} only matches via the grader list branch). Child JSON/pagination work will need to pick one shape.
| match: {json: true} | ||
| expect_output: | ||
| db_path: /fixtures/imported.db | ||
| focus_updated: true |
There was a problem hiding this comment.
Low:与只读沙箱声明存在契约张力。
suite 声明 project: read_only、fixtures: read_only 且禁止 pt_snap.focus_write / filesystem.write(suite.yaml:74-75,87,91),但本用例要求 focus_updated: true 且发布路径在 /fixtures/ 下。真实 import 发布会原子写入项目内 .pt-snap/focus.json,严格执行沙箱的 runner 无法复现该 target 轨迹。当前无人强制执行,属潜在冲突。建议在契约中说明 pt_snap.import 为原子例外,或去掉强制 focus_updated: true。
There was a problem hiding this comment.
Agreed; tracking as follow-up, no code in this PR. The suite declares project/fixtures: read_only and forbids pt_snap.focus_write / filesystem.write, but import-success requires focus_updated: true. Nothing enforces the sandbox today; the runner or contract should mark pt_snap.import as an atomic exception when that path is implemented.
| assert by_id["missing-database"][1].error_conclusion is True | ||
|
|
||
|
|
||
| def test_error_status_actions_match_structured_failures() -> None: |
There was a problem hiding this comment.
Low:状态相等的负方向缺少测试。
grader.py:121 的状态相等目前只有正方向测试;没有用例固定“status: error 动作不得匹配成功调用”(反向亦然)——这是防止幻觉重试拿满分的承重点。建议每个方向加一个合成不匹配的 grader 单测。
There was a problem hiding this comment.
Agreed; tracking as follow-up, no code in this PR. Status equality still only has the positive-direction test. A synthetic mismatch in each direction (error action vs success call, and the reverse) belongs in a later harness test, not this baseline freeze.
| assert payload == printed | ||
| assert payload["task_success_rate"] == 0.0 | ||
| assert payload["error_conclusion_rate"] == 0.8571 | ||
| assert [case["case_id"] for case in payload["cases"]] == list(CASE_IDS) |
There was a problem hiding this comment.
Low:mean_output_bytes 无断言,另有未测路径。
mean_output_bytes 是 PR 头号对比指标(metrics.py:110),但全仓测试均无断言;run_output_bytes(metrics.py:16-26)、空输入 ValueError(metrics.py:101)、missing run records 退出(__main__.py:55)也未被覆盖。建议补一个 mean_output_bytes 断言(或固定每案 output_bytes)。
There was a problem hiding this comment.
Partially addressed in faee29f: test_baseline_cli_writes_metrics now asserts mean_output_bytes against the per-case output_bytes mean. Empty-summary ValueError, the missing run records exit, and run_output_bytes itself remain untested and are follow-up.
Documented baseline --output paths live under missing parents, and the target recovery trace should follow leak_detection's size DESC order.
📝 Description
Add the #136 Agent CLI end-to-end evaluation baseline on the existing
tests/skillsharness. This records current CLI-only Agent behavior so later JSON, helper, pagination, and related work can be compared against a frozen score. Product CLI/API/MCP behavior is unchanged.The suite covers the seven parent-issue scenarios: help/helper discovery, structured template-error recovery, explicit db/device targeting, truncated queries, unconfirmed live blocks, missing-database failures, and structured import results.
🔗 Related Issue
Related to #136 (evaluation baseline only; P0/P1 product work remains on child issues).
🧪 Testing
Tests added/updated
Manual testing completed
PYTHONPATH=src pytest tests/skills— 80 passedPYTHONPATH=src pytest— 875 passed, 1 xfailedruff check tests/skillsandblack --check tests/skillspython -m tests.skills baseline tests/skills/suites/pt-snap-agent-e2e/suite.yaml tests/skills/suites/pt-snap-agent-e2e/baselines/pre-changePre-change metrics: task success 0.0, mean calls 1.0, mean output ~101 bytes, error-conclusion 0.8571. Target traces score 1.0 / 0.0.
✅ Checklist
Snapshot Runtime Provenance
Snapshot provenance decision: no-update
Snapshot provenance no-update reason: This PR only changes the tests/skills evaluation harness, descriptors, and recorded baselines; it does not touch src/pt_snap_cli/snapshot.
📋 Additional Notes
Do not close #136 with this PR. Child issues (#137–#142, plus tracked #120) still own the product changes. Re-run the target baseline after those land.