Skip to content

Add CLI/MCP contract coverage - #64

Merged
codeasier merged 2 commits into
mainfrom
issue-61-contract-tests
Jun 7, 2026
Merged

codeasier merged 2 commits into
mainfrom
issue-61-contract-tests

Conversation

@codeasier

@codeasier codeasier commented Jun 7, 2026 •

Copy link
Copy Markdown
Owner

📝 Description

This PR adds a CLI/MCP semantic contract test suite so future changes to shared snapshot capabilities fail in CI when the two adapters drift apart.

The goal is to make pt-snap CLI and pt-snap-mcp continue to represent the same underlying behavior as the shared API/core layer evolves, especially around focus handling, template metadata, query execution, and shared error cases.

🔗 Related Issue

Fixes #61

Why this change

The project already moved core behavior into shared services, but that alone does not prevent adapter drift:

  • CLI behavior can change without MCP semantics staying aligned.
  • MCP responses can evolve independently from CLI-facing behavior.
  • New shared capabilities can land with only one-side tests.
  • Error semantics and metadata can silently diverge while both surfaces still “work”.

This PR adds CI-visible contract checks so that kind of drift fails early.

What changed

  • Added tests/test_contract_cli_mcp.py
    • compares CLI and MCP semantics for the same fixture and same inputs
    • covers focus handling
    • covers template list semantics
    • covers template info semantics
    • covers query execution semantics
    • covers missing-focus and missing-template error semantics
  • Updated pt-snap query --template-info output to include:
    • Category
    • Devices
      so CLI metadata now exposes the same shared fields already available through the API/MCP path.
  • Updated README.md
    • documents the CLI/MCP semantic contract rule
    • states that new shared CLI/MCP capabilities must add a contract case

🧪 Testing

  • Tests added/updated
  • Manual testing completed

Verification run locally in the issue worktree:

  • conda run -n cc ruff check .
  • conda run -n cc pytest → 292 passed

✅ Checklist

  • Code follows project guidelines
  • Documentation updated (if applicable)
  • No new warnings introduced
  • Changes tested locally

📸 Screenshots (if applicable)

📋 Additional Notes

The contract tests intentionally normalize CLI text output before comparing it with MCP results, so the suite checks semantic parity rather than raw formatting identity.

codeasier added 2 commits June 7, 2026 21:24
Add cross-surface contract tests for focus, template metadata, query execution, and shared error semantics so drift between the CLI and MCP adapters fails in CI. Update template info output and document the contract-testing rule for new shared capabilities.
Apply Black formatting to the new CLI/MCP contract test file so the lint and format CI job passes.
@codeasier
codeasier merged commit 335c977 into main Jun 7, 2026
5 checks passed
@codeasier
codeasier deleted the issue-61-contract-tests branch June 7, 2026 14:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE] Add contract tests to prevent CLI and MCP semantic drift

1 participant