Skip to content

feat(mcp): declare all four behavior hints on every tool - #3395

Merged
kodiakhq[bot] merged 4 commits into
mainfrom
agent/mcp-tool-annotations
Oct 19, 2026
Merged

kodiakhq[bot] merged 4 commits into
mainfrom
agent/mcp-tool-annotations

Conversation

@brandon-pereira

Copy link
Copy Markdown
Member

Summary

MCP compatibility checks fail any tool that does not declare readOnlyHint, destructiveHint, idempotentHint, and openWorldHint as booleans. Each tool set only one hint before this change.

  • Adds presets in packages/api/src/mcp/tools/annotations.ts:
    • READ_ONLY_TOOL for get/list/query tools.
    • UPSERT_TOOL for create-or-update tools. They are not idempotent: omitting id creates a new object on every call.
    • IDEMPOTENT_DESTRUCTIVE_TOOL for delete_* and patch_dashboard.
  • openWorldHint is false on every tool, because tools only touch the team's own Mongo metadata and ClickHouse data.
  • RegisterToolFn now requires all four hints, so a new tool without them fails to compile.
  • Test: packages/api/src/mcp/__tests__/toolAnnotations.test.ts.

Downstream (EE) impact: the EE-only tools clickstack_get_source_filter_values, clickstack_backtest_anomaly_alert, and clickstack_list_teams will fail typecheck on the upstream merge until they get annotations.

How to reproduce

Starting point: local full stack (yarn dev) with npx @modelcontextprotocol/inspector connected to the local MCP endpoint (see MCP.md).

On main:

  1. Run initialize, then tools/list.
  2. Each tool's annotations has only one hint set.

On this branch:

  1. Run initialize, then tools/list.
  2. Every tool's annotations has all four hints set as booleans.

How to test on Vercel preview

N/A — non-UI change.

References

  • Related PRs: agent/mcp-tool-descriptions (independent; touches neighboring lines, but the two 3-way merge cleanly)

@changeset-bot

changeset-bot Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: dc4bc54

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages
Name Type
@hyperdx/api Patch
@hyperdx/app Patch
@hyperdx/otel-collector Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercel Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Oct 19, 2026 5:06pm UTC
hyperdx-storybook Ready Ready Preview Oct 19, 2026 5:06pm UTC

Request Review

@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Duplicate code analysis

No new duplicate code introduced by this PR.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 124 in total (1.67%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

@greptile-apps

greptile-apps Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Low impact] The PR appears safe to merge with the documented downstream EE updates.

Summary

Adds shared presets declaring all four MCP behavior hints, requires them in RegisterToolFn, and adds a catalog-wide test.

  • Every MCP tool now lists all four behavior hints.

Reviews (4) · Last reviewed commit: "Merge branch 'main' into agent/mcp-tool-..." · Reviewed by Greptile

Comment thread packages/api/src/mcp/tools/query/sql.ts Resolved
@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 436 passed • 1 skipped • 1455s

Status Count
✅ Passed 436
❌ Failed 0
⚠️ Flaky 2
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

@brandon-pereira
brandon-pereira marked this pull request as ready for review October 16, 2026 19:41
@github-actions github-actions Bot added the review/tier-2 Low risk — AI review + quick human skim label Oct 16, 2026
@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

🔵 Tier 2 — Low Risk

Small, isolated change with no API route or data model modifications.

Why this tier:

  • Standard feature/fix — introduces new logic or modifies core functionality

Review process: AI review + quick human skim (target: 5–15 min). Reviewer validates AI assessment and checks for domain-specific concerns.
SLA: Resolve within 4 business hours.

Stats
  • Production files changed: 32
  • Production lines changed: 142 (+ 26 in test files, excluded from tier calculation)
  • Branch: agent/mcp-tool-annotations
  • Author: brandon-pereira

To override this classification, remove the review/tier-2 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 critical or 🟠 major in code this PR adds or changes: fix before asking for review.
  • 🔵 minor in code this PR changes: your call. Fix if small, otherwise reply.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    severity. Reply with one sentence; if it is critical, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

1 finding(s): 🔴 0 critical · 🟠 0 major · 🔵 1 minor

None could be anchored to changed lines; they are listed below.

Findings outside the changed lines

1 minor
  • 🔵 packages/api/src/mcp/__tests__/toolAnnotations.test.ts:30 — Test only re-checks what the compiler already enforces, not which preset each tool gets → RegisterToolFn in packages/api/src/mcp/tools/types.ts:49 already makes all four hints required booleans, so a tool with a missing hint won't compile. The test passes even if a delete or save tool is given READ_ONLY_TOOL, which would tell clients a destructive call is safe to auto-approve. Add assertions that pin the mapping, for example: every clickstack_delete_* and clickstack_save_* tool has readOnlyHint: false, destructiveHint: true, and the save_* tools have idempotentHint: false. A per-tool snapshot of the annotations would also work.

Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides.

@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

✅ No critical issues found.

This PR is a mechanical, low-risk change: it introduces three shared annotation presets (READ_ONLY_TOOL, UPSERT_TOOL, IDEMPOTENT_DESTRUCTIVE_TOOL), tightens RegisterToolFn so every tool must declare all four behavior hints, swaps each tool onto the correct preset, and adds a test asserting the hints are present as booleans. The preset-to-tool mapping is semantically consistent (read tools → read-only, save_* → non-idempotent upsert, delete_*/patch_dashboard → idempotent destructive), the type-level enforcement achieves its stated compile-time guarantee, and no behavior changes beyond the advertised tools/list metadata. The findings below are advisory accuracy and coverage concerns, not blockers.

🟡 P2 -- recommended

  • packages/api/src/mcp/tools/query/sql.ts:82 -- clickstack_sql now newly asserts openWorldHint: false, but raw ClickHouse SQL under readonly=2 blocks writes, not external reads via table functions such as url(), s3(), remote(), or mysql(), so the advertised scope understates what the tool can reach.
    • Fix: Advertise openWorldHint: true for clickstack_sql (or restrict external table functions before asserting closed-world) so the hint matches the tool's real reach.
    • security, api-contract, correctness, previous-comments
  • packages/api/src/mcp/__tests__/toolAnnotations.test.ts:13 -- the test only verifies each hint is a boolean, so a tool wired to the wrong preset (e.g. a destructive tool advertising readOnlyHint: true) would still pass, leaving the central correctness goal of the change unguarded.
    • Fix: Add assertions that representative read, upsert, and delete tools report their expected hint values, not merely that the four keys are booleans.
    • testing, correctness
🔵 P3 nitpicks (1)
  • packages/api/src/mcp/tools/annotations.ts:14 -- the blanket comment stating every tool "reads or writes only the calling team's own ClickStack metadata (MongoDB) and telemetry (ClickHouse), so none of them is open-world" is contradicted by the raw-SQL path and will mislead future maintainers reusing these presets.
    • Fix: Narrow the comment to note that raw-SQL execution is the exception, or scope the claim to the builder/metadata tools it accurately describes.

Reviewers (7): correctness, testing, maintainability, kieran-typescript, api-contract, security, previous-comments.

Testing gaps:

  • No test verifies that specific tools map to the correct preset (hint values), only that all four hints exist as booleans — a wrong-preset regression would pass CI.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

@brandon-pereira
brandon-pereira requested review from a team and jordan-simonovski and removed request for a team October 16, 2026 19:52

@jordan-simonovski jordan-simonovski left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved with one possible improvement we could make

* Create-or-update tools: omitting `id` creates a new object on every call
* (not idempotent), and providing `id` overwrites the stored object.
*/
export const UPSERT_TOOL: ToolBehaviorAnnotations = {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OOC could we also add a preset for writes that aren't destructive?
Stuff like add_investigation_note only appends, so it isn't destructive or idempotent, and share_notebook is idempotent but not destructive. Without a matching preset, EE would either hand-write the objects or pick UPSERT_TOOL. Would something like this cover both?

export const APPEND_TOOL: ToolBehaviorAnnotations = {
  readOnlyHint: false,
  destructiveHint: false,
  idempotentHint: false,
  openWorldHint: false,
};

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, will handle this in EE for now

@kodiakhq
kodiakhq Bot merged commit e494a46 into main Oct 19, 2026
29 checks passed
@kodiakhq
kodiakhq Bot deleted the agent/mcp-tool-annotations branch October 19, 2026 17:12

This branch was successfully deployed

2 active deployments
Preview – hyperdx-oss — dc4bc542 Deployed Oct 19, 2026 by vercel[bot]
Preview – hyperdx-storybook — dc4bc542 Deployed Oct 19, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

automerge review/tier-2 Low risk — AI review + quick human skim

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants