Skip to content

feat(tests): add Agent CLI end-to-end evaluation baseline - #145

Merged
codeasier merged 2 commits into
mainfrom
feat/issue-136-agent-e2e-baseline
Sep 18, 2026
Merged

codeasier merged 2 commits into
mainfrom
feat/issue-136-agent-e2e-baseline

Conversation

@codeasier

Copy link
Copy Markdown
Owner

📝 Description

Add the #136 Agent CLI end-to-end evaluation baseline on the existing tests/skills harness. This records current CLI-only Agent behavior so later JSON, helper, pagination, and related work can be compared against a frozen score. Product CLI/API/MCP behavior is unchanged.

The suite covers the seven parent-issue scenarios: help/helper discovery, structured template-error recovery, explicit db/device targeting, truncated queries, unconfirmed live blocks, missing-database failures, and structured import results.

🔗 Related Issue

Related to #136 (evaluation baseline only; P0/P1 product work remains on child issues).

🧪 Testing

  • Tests added/updated

  • Manual testing completed

  • PYTHONPATH=src pytest tests/skills — 80 passed

  • PYTHONPATH=src pytest — 875 passed, 1 xfailed

  • ruff check tests/skills and black --check tests/skills

  • python -m tests.skills baseline tests/skills/suites/pt-snap-agent-e2e/suite.yaml tests/skills/suites/pt-snap-agent-e2e/baselines/pre-change

Pre-change metrics: task success 0.0, mean calls 1.0, mean output ~101 bytes, error-conclusion 0.8571. Target traces score 1.0 / 0.0.

✅ Checklist

  • Code follows project guidelines
  • Documentation updated (if applicable)
  • No new warnings introduced
  • Changes tested locally

Snapshot Runtime Provenance

  • Snapshot provenance evaluation completed when snapshot runtime files change
    Snapshot provenance decision: no-update
    Snapshot provenance no-update reason: This PR only changes the tests/skills evaluation harness, descriptors, and recorded baselines; it does not touch src/pt_snap_cli/snapshot.

📋 Additional Notes

Do not close #136 with this PR. Child issues (#137–#142, plus tracked #120) still own the product changes. Re-run the target baseline after those land.

Freeze #136's pre-change Agent metrics so later JSON, helper, and
pagination work can be compared against a recorded CLI-only score.

@codeasier codeasier left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

交叉评审结论:Approve with low-severity follow-ups。

  • 范围确认:仅测试 harness(tests/skills/*),无产品代码改动;评分与指标数字已独立复算一致(target 7/7 满分 100;pre-change 0/7,error_conclusion_rate 0.8571,mean_call_count 1.0)。
  • 建议合并前修复:#1(--output 父目录)、#2(基线行序)。
  • 可跟踪为后续工作:#3(gateway 错误形状)、#4(信封契约)、#5(沙箱张力)、#6(测试缺口)。

评审溯源:3 位独立评审(qwen3.8-flash、deepseek-v4.1-flash、muse-spark-1.3-contributor,3/3 成功)+ 父会话逐条对照快照验证。

Comment thread tests/skills/__main__.py Resolved
"template": "leak_detection"
},
"status": "error",
"output": {

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low:该错误形状无法由 RecordingToolGateway 产生。

status: "error" 且带结构化 output 是恢复/缺失目标用例的要求,但 harness/gateway.py:48-63 的异常路径只记 error 字符串(output 保持 None),从不同时记录两者。未来基于该 gateway 的 local-agent runner 会无法通过这些用例。建议明确约定或扩展 gateway 以支持结构化错误负载(wrong-template-recovery.json:11-16 同样如此)。

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed; tracking as follow-up, no code in this PR. The target traces describe the #138 structured-error envelope. RecordingToolGateway still records exceptions as an error string with output=None, so a future local-agent runner cannot satisfy these cases until the gateway (or JSON error protocol) lands.

| `pt_snap.help` | `{mentions_helper: true, helper_skill: "pt-snap-helper"}` when help points at the helper skill |
| `pt_snap.skill_list` | JSON listing; `json: true` in arguments |
| failed `pt_snap.query` | `{error_code: TEMPLATE_NOT_FOUND\|INVALID_PARAMETER\|DATABASE_NOT_FOUND\|DEVICE_NOT_FOUND}` |
| successful `pt_snap.query` | envelope with `db_path`, `device_id`, `returned`, `has_more`, and `rows` |

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low:信封契约声明与两个 target 基线矛盾。

此处称成功的 pt_snap.query 返回含 db_path、device_id、returned、has_more、rows 的信封;但 truncated-query.json:14-21 缺 db_path/device_id,live-blocks-unconfirmed.json:13-20 是裸列表。live-blocks 用例的 expect_output: {size: 4096} 目前仅靠 _output_contains 的列表分支(grader.py:96-97)通过;若是信封 dict 反而会失败(dict 匹配仅顶层,grader.py:94-95)。子任务 #137–#142 落地时二者必改其一。建议先在契约中限定适用范围,或把两个 target 基线归一化为完整信封。

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed; tracking as follow-up, no code in this PR. The contract describes the intended success envelope, while truncated-query omits db_path/device_id and live-blocks-unconfirmed is a bare list (so {size: 4096} only matches via the grader list branch). Child JSON/pagination work will need to pick one shape.

match: {json: true}
expect_output:
db_path: /fixtures/imported.db
focus_updated: true

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low:与只读沙箱声明存在契约张力。

suite 声明 project: read_only、fixtures: read_only 且禁止 pt_snap.focus_write / filesystem.write(suite.yaml:74-75,87,91),但本用例要求 focus_updated: true 且发布路径在 /fixtures/ 下。真实 import 发布会原子写入项目内 .pt-snap/focus.json,严格执行沙箱的 runner 无法复现该 target 轨迹。当前无人强制执行,属潜在冲突。建议在契约中说明 pt_snap.import 为原子例外,或去掉强制 focus_updated: true。

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed; tracking as follow-up, no code in this PR. The suite declares project/fixtures: read_only and forbids pt_snap.focus_write / filesystem.write, but import-success requires focus_updated: true. Nothing enforces the sandbox today; the runner or contract should mark pt_snap.import as an atomic exception when that path is implemented.

assert by_id["missing-database"][1].error_conclusion is True


def test_error_status_actions_match_structured_failures() -> None:

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low:状态相等的负方向缺少测试。

grader.py:121 的状态相等目前只有正方向测试;没有用例固定“status: error 动作不得匹配成功调用”(反向亦然)——这是防止幻觉重试拿满分的承重点。建议每个方向加一个合成不匹配的 grader 单测。

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed; tracking as follow-up, no code in this PR. Status equality still only has the positive-direction test. A synthetic mismatch in each direction (error action vs success call, and the reverse) belongs in a later harness test, not this baseline freeze.

assert payload == printed
assert payload["task_success_rate"] == 0.0
assert payload["error_conclusion_rate"] == 0.8571
assert [case["case_id"] for case in payload["cases"]] == list(CASE_IDS)

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low:mean_output_bytes 无断言,另有未测路径。

mean_output_bytes 是 PR 头号对比指标(metrics.py:110),但全仓测试均无断言;run_output_bytes(metrics.py:16-26)、空输入 ValueError(metrics.py:101)、missing run records 退出(__main__.py:55)也未被覆盖。建议补一个 mean_output_bytes 断言(或固定每案 output_bytes)。

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partially addressed in faee29f: test_baseline_cli_writes_metrics now asserts mean_output_bytes against the per-case output_bytes mean. Empty-summary ValueError, the missing run records exit, and run_output_bytes itself remain untested and are follow-up.

Documented baseline --output paths live under missing parents, and the
target recovery trace should follow leak_detection's size DESC order.
@codeasier
codeasier merged commit d73013d into main Sep 18, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE] Agent 友好化改造:全命令 --json + pt-snap-helper 引导技能 + 移除 MCP

1 participant