Skip to content

feat: add explainable MCP tool-surface policy review - #8

Merged
DivyamTalwar merged 3 commits into
mainfrom
codex/orrery-tool-surface-review
Sep 5, 2026
Merged

DivyamTalwar merged 3 commits into
mainfrom
codex/orrery-tool-surface-review

Conversation

@DivyamTalwar

Copy link
Copy Markdown
Owner

Summary

Adds a human-readable policy layer beside Orrery's authoritative full-contract digest:

  • checks in a deterministic projection of tool names, accepted arguments, and all MCP behavior annotations
  • classifies added arguments/stateful tools, readonly downgrades, destructive/open-world changes, removals, and description/schema-only digest changes
  • makes bun run validate fail if the reviewed policy snapshot drifts
  • adds bun run tools:review for a structured reapproval report
  • documents that annotations are review signals, not host enforcement

This PR is independently based on e8a9604c8ac2b15de76d48080604c9151e76ec1d.

Competitive rationale and provenance

Cross-Code Organizer hashes discovered MCP tool definitions, while MCP Hub tracks changing server capabilities. The MCP tools specification defines behavior annotations but warns clients to treat server-provided annotations as untrusted. Orrery already blocks contract-digest changes; the missing review surface was explaining whether the delta expands permissions.

The comparator and severity model are original Orrery code. No competitor implementation was copied. The complete digest remains authoritative; the readable policy is deliberately a projection, not a replacement.

Verification

  • deterministic projection test
  • critical readonly-downgrade + new-argument classification test
  • description/schema-only digest-change test
  • unchanged-baseline no-noise test
  • baseline parity enforced by repository validation and packaged CI
  • authoritative Linux/macOS CI is required before review

Evidence limit

The review classifies declared contracts. It cannot prove a server is honest or a host enforces an annotation, and the documentation states that boundary explicitly.

Copy link
Copy Markdown
Owner Author

Verification evidence

Latest head: f679c22f81c35be09149b6102b395582b5afbb8c

  • CI run 32663094205, job 97252109361: success — frozen install + full bun run ci, including policy-baseline validation
  • Compatibility run 32663094252: success
    • Ubuntu job 97252109682: typecheck + complete test suite
    • macOS job 97252109598: typecheck + complete test suite
  • Focused evidence: 4 new review tests cover deterministic projection, readonly/argument expansions, content-only contract drift, and an unchanged baseline.

MCP annotations remain untrusted declarations; this PR makes their change reviewable and blocking, not magically enforceable.

Copy link
Copy Markdown
Owner Author

Competitive-review validation

Latest head f679c22f81c35be09149b6102b395582b5afbb8c is green: CI 32663094205; Compatibility 32663094252 on Ubuntu and macOS.

The human-readable tool-contract projection responds to MCP management/security patterns in Cross-Code Organizer and the current MCP tools specification. ORRERY retains its complete canonical digest as the authority; annotations are review signals, never trusted enforcement. No third-party code was copied.

Per the requested workflow, this PR remains open and must not be merged.

@DivyamTalwar

Copy link
Copy Markdown
Owner Author

Merge-order note — conflicts with #7.

Both this PR and #7 rewrite the same "test" line in package.json to append their own test file. Each is clean against main on its own, so GitHub reports both as mergeable — the conflict only appears after the first one lands.

The dangerous resolution is the quiet one: keep one test file, drop the other, and CI stays green because the missing tests simply never run.

Whichever merges second needs its test script to list both files:

tools/capability-planner.test.ts tools/tool-surface-review.test.ts

Ping me and I'll rebase the second one.

@DivyamTalwar
DivyamTalwar merged commit a5144f3 into main Sep 5, 2026
4 checks passed
@DivyamTalwar
DivyamTalwar deleted the codex/orrery-tool-surface-review branch September 5, 2026 08:25
DivyamTalwar added a commit that referenced this pull request Sep 5, 2026
Problem
Orrery gained three read-only diagnostics in three separate pull requests — the
capability planner (#7), the tool-surface review (#8), and the semantic
compatibility doctor (#9). Each arrived with a sentence appended wherever its own
change happened to touch the README, so two of them ended up as orphan paragraphs
below the development gate and the third existed only as a comment inside a code
block.

Read individually they look like three unrelated utilities. They are not: they are
the same capability at three different moments — before you choose a client, after
you install, and whenever the tool surface moves. That framing was nowhere on the
page, so the most checkable thing about this project was also the least visible.

Approach
Add "Interrogate it yourself" between "What it will not do" and "When not to use
this", so the honesty block reads in one run: what it refuses to do, how you can
check that for yourself, and when you should not use it at all.

The section is organised by moment rather than by tool, and each entry leads with
what the command *refuses* to do, because that is the part with value. A path-only
portability matrix would report six clients as working; the planner names the ones
that would quietly downgrade advisor isolation and refuses them. The doctor
separates what it observed on disk from what it cannot know about runtime
behaviour. The review turns "the digest moved" into whether the thing that moved
can take a new argument, hold state, or has dropped a `readOnlyHint`.

It closes by inverting the usual precedence: if a diagnostic disagrees with the
README, the README is wrong.

Verification
Every documented invocation was run, not copied from a pull request description.
`plan` with the documented flags returns `bestFit: codex` and refuses cursor and
the remaining clients. `tools:review` returns `changed: false`,
`permissionExpansion: false`. `doctor` reports schema 2 with `readOnlyMechanism`
and per-role adapter state. `bun run ci` is green.

Impact
Documentation only.
DivyamTalwar added a commit that referenced this pull request Sep 5, 2026
Problem
Four pull requests merged into main without a changelog entry between them: the
semantic compatibility doctor (#9), the explainable tool-surface review (#8), the
capability planner (#7), and the toolchain bump (#15). The Unreleased section
described only the Astra lane, so the three most user-visible additions since 0.6.0
existed in the code and in the README but nowhere in the release record.

A changelog that silently omits four merges is worse than no changelog, because it
is read as complete.

Approach
Document all four, then cut Unreleased as 0.7.0 and bring the three declared
versions into agreement. Minor rather than patch: the release adds a second native
implementation lane and three read-only diagnostics. Nothing is removed and no
interface breaks, so it is not a major.

Each entry records what the thing refuses to do, since that is the part with value:
the planner refuses hosts that would silently weaken requested effort or advisor
isolation; the review turns "the digest moved" into whether the change is
permission-bearing; the doctor no longer reports a marker-only or weakened adapter
as healthy. The toolchain entry records why the bump was applied by hand, so the
next person does not reopen the Dependabot question.

Verification
`bun run ci` is green: verify.sh three-role checks pass, 82 tests across 6 files,
validate, sbom (now emitting orrery-0.7.0.cdx.json, 18 packaged files, 0 runtime
dependencies), and release-check. `bun run tag:check -- v0.7.0` passes, so the tag
and all three manifests agree.

Impact
The release record now matches what shipped. Ready to tag v0.7.0 once merged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant