Skip to content

feat(agent): complete #136 CLI acceptance - #157

Merged
codeasier merged 4 commits into
mainfrom
issue-136-agent-e2e-acceptance
Sep 24, 2026
Merged

codeasier merged 4 commits into
mainfrom
issue-136-agent-e2e-acceptance

Conversation

@codeasier

Copy link
Copy Markdown
Owner

Description

Complete the remaining implementation and local acceptance work for the Agent-first CLI contract:

  • extend the E2E suite to parameter-error recovery, databases without devices, competing databases, template semantics, and the real import focus envelope;
  • retain structured CLI error payloads and match nested JSON rows deterministically;
  • make helper onboarding explicit in root help, including installation and restart;
  • replace the basic wheel smoke with a clean-venv Agent acceptance that installs the bundled helper and completes trusted import, capabilities, overview, and peak query using structured output;
  • make JSON import output clean by suppressing runtime log records during the JSON operation.

Post-change evaluation improves task success from 0.0 to 1.0 and error-conclusion rate from 0.8 to 0.0. Structured recovery raises mean calls from 1.0 to 1.4 and mean recorded output from 110.2 to 295.8 bytes; regression tests enforce explicit budgets of 1.5 calls and 4096 bytes.

Related Issue

Refs #136. Keep the issue open until this change is published because the public pip install pt-snap-cli package is still 0.3.0.

Testing

  • Tests added/updated
  • Manual testing completed
  • pytest — 1024 passed, 1 xfailed
  • ruff check .
  • black --check .
  • python -m basedpyright --pythonpath "$(python -c 'import sys; print(sys.executable)')" — 0 errors
  • Agent E2E suite validation and pre-/post-change baseline grading
  • Trusted fixture provenance test
  • Clean-wheel Agent acceptance in an isolated venv

Checklist

  • Code follows project guidelines
  • Documentation updated
  • No new warnings introduced
  • Changes tested locally

Snapshot Runtime Provenance

  • Snapshot provenance evaluation completed when snapshot runtime files change
    Snapshot provenance decision: no-update
    Snapshot provenance no-update reason: No files under src/pt_snap_cli/snapshot/ changed; this PR only changes the CLI adapter, evaluation harness, tests, documentation, and CI packaging gate.

Additional Notes

The clean-wheel check uses the provenance-approved trusted snapshot only for packaging acceptance. The declarative E2E suite continues to use synthetic SQLite fixtures and never materializes pickle input.

@codeasier codeasier left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cross-Review 结论(3 模型独立评审 + 父会话逐条复核)

对 head 0f9b693(merge-base 55f7511)完成三方独立模型评审,所有候选发现均已对照快照代码复核。

产出代码本身未发现正确性 / 安全缺陷:_import_snapshot_result 的 logging.disable 作用域与 finally 恢复正确;JSON envelope、AGENT_HELP_EPILOG、descriptor 挂载/输出校验、no-device → DEVICE_NOT_FOUND 映射、gateway 结构化错误路径均与验收脚本预期一致。

行内评论 4 条:

严重度 位置 摘要
Medium .github/scripts/clean_wheel_agent_acceptance.py:147 验收 venv 非封闭,wheel 依赖闭包未验证,editable 安装可能引起假失败
Medium src/pt_snap_cli/cli.py:186 stderr 抑制(本 PR 唯一产出代码变更)无快速回归测试
Low tests/skills/harness/grader.py:102 expect_output 列表按位置匹配,文档未说明
Low tests/skills/harness/fixtures.py:60 devices 非空校验全局放宽,意外空 devices: 不再 fail fast

另有 1 条候选发现(logging.disable 进程级全局、非线程安全)经复核驳回:单模型报告,且无实际失败路径(单发 Typer 进程、finally 恢复)。

— 独立评审:volceapi/qwen3.8-flash、opencode-go/deepseek-v4.1-flash、opencode-go/muse-spark-1.3-contributor;裁决:opencode 父会话

home.mkdir()
cwd.mkdir()
output_dir.mkdir()
venv.EnvBuilder(with_pip=True, system_site_packages=True, clear=True).create(venv_dir)

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Medium] 验收 venv 并不 "clean":依赖闭包未经验证

system_site_packages=True 使 venv 继承宿主 site-packages,而 3.13 job 先在基础解释器执行 pip install -e ".[dev]"(.github/workflows/test.yml:98),随后此处又以 --no-deps --no-index 安装 wheel(本文件 :152-166)。后果:

  • wheel 声明的运行时依赖(typer/click/jinja2/pyyaml)全部继承自宿主——wheel 缺依赖或依赖钉错照样通过本门禁;
  • import pt_snap_cli 解析到 wheel 还是宿主的 editable 安装取决于 PEP 660 editable 模式(meta-path finder vs .pth),_is_inside 守卫(:181-191、:240-243)可能把 sys.path/sys.meta_path 差异变成 release gate 的假失败。

建议:改用 system_site_packages=False 并以本地依赖 wheelhouse(--find-links)安装依赖;或显式断言模块解析路径位于 venv 自身 site-packages 并屏蔽继承的 editable finder。

来源:3/3 独立评审模型一致报告,已在 head 0f9b693 快照复核确认。

Comment thread src/pt_snap_cli/cli.py
if not json_output:
return ImportService().import_snapshot(options)
previous_disable = logging.root.manager.disable
logging.disable(logging.CRITICAL)

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Medium] 本 PR 唯一的产出代码行为变更缺少快速回归测试

logging.disable(CRITICAL) 用于抑制 snapshot runtime 写向 stderr 的进度日志(snapshot/util/logger.py:125 的 StreamHandler() 默认 stderr),但现有测试无法守住它:

  • _assert_clean_json_success(tests/test_cli_json.py:153-161)只校验 stdout,从不断言 result.stderr 为空;
  • test_import_json_fresh_and_reuse(tests/test_cli_json.py:397-420)继承同一缺口。

直接回退此处的抑制,pytest 仍然全绿,只有 3.13-only 的 clean-wheel 门禁才会失败(慢、需先构建 wheel)。建议补充针对 import --json(fresh + focused 路径)的空 stderr 断言,并校验 logging.root.manager.disable 在调用后恢复原值。

and len(expected) <= len(actual)
and all(
_value_contains(item, candidate)
for item, candidate in zip(expected, actual, strict=False)

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Low] expect_output 列表匹配是按位置的前缀匹配,文档未说明

zip(expected, actual, strict=False) 要求期望行按索引与实际行对齐:行集合相同但顺序不同的查询结果会被判为不匹配。当前基线是确定性的,属潜在风险(基线若从 live CLI 输出重新生成且字段顺序变化会静默失败)。tests/skills/README.md:25-26 只说 "matched recursively",未提及顺序敏感。建议:在 README 注明位置匹配语义,或将行列表改为无序子集匹配(每个期望行在未使用的实际行中找到即可)。

raise DescriptorError(f"{path} must declare synthetic-snapshotdb schema version 1")
if not isinstance(data["devices"], dict) or not data["devices"]:
raise DescriptorError(f"{path}: devices must be a non-empty mapping")
if not isinstance(data["devices"], dict):

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Low] devices 非空校验对所有 fixture 放宽,而非仅限新增的空场景

移除 or not data["devices"] 后,任何 synthetic fixture 里意外写空的 devices: 都会静默构建出无设备的 SnapshotDB,而不是 fail fast。实际只有 tests/skills/suites/pt-snap-agent-e2e/fixtures/no-devices.yaml 需要该行为。建议:默认仍要求非空映射,为空设备场景增加显式 opt-in(如 fixture 级 allow_empty_devices 标志)。

@codeasier
codeasier merged commit 4c4d490 into main Sep 24, 2026
6 checks passed
@codeasier
codeasier deleted the issue-136-agent-e2e-acceptance branch September 24, 2026 02:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant