Skip to content

feat(mcp): run multiple clickstack_sql queries in parallel - #3421

Merged
kodiakhq[bot] merged 6 commits into
mainfrom
brandon/mcp-sql-batch-queries
Nov 3, 2026
Merged

kodiakhq[bot] merged 6 commits into
mainfrom
brandon/mcp-sql-batch-queries

Conversation

@brandon-pereira

@brandon-pereira brandon-pereira commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

Summary

Agents investigating an incident often need several independent raw SQL queries (compare two sources, a few breakdowns of the same window). Today each one is a separate clickstack_sql round trip. This adds a queries array so one call runs up to 10 of them in parallel.

  • clickstack_sql takes either sql (unchanged) or queries: [{ id?, sql, connectionId?, sourceId? }]. Queries share startTime/endTime and default to the top-level connectionId/sourceId.
  • Queries run 5 at a time under one 60s deadline for the whole call. Queued queries that reach the deadline are never sent to ClickHouse.
  • Each result reports ok, error, or timeout, in input order, with its index and optional id. One bad query doesn't fail the call; isError is set only when every query fails.
  • The 100k-character response budget is split across the batch (at most 50k per query, the current single-query limit).
  • The existing per-query checks (dashboard-only macros, source not found, connection mismatch, $__sourceTable without a source) moved into prepareSql.ts with no behavior change. Source lookups are cached per call.
  • withDeadline moved from queryTiles.ts to mcp/utils/timeout.ts so both batch tools share it.
  • Observability: mcp.sql.mode and mcp.sql.batch.* span attributes, plus a hyperdx.mcp.sql.batch.queries counter tagged by status.

Schema change to note: sql is no longer in the tool's required list, since sql and queries are alternatives. The handler rejects calls that pass both or neither.

We looked at MCP tasks for this and decided against them for now: the SDK API is experimental, and the 2026-07-28 spec moved tasks into a separate, redesigned extension that the v2 SDK doesn't implement on the server.

How to reproduce

Starting point: yarn dev, then connect an MCP client (e.g. yarn dev:mcp in packages/api for the inspector) with a personal API key.

On main:

  1. Call clickstack_sql with queries: [{ "sql": "SELECT 1" }] and any connectionId. The call fails because sql is required and queries is unknown.

On this branch:

  1. Call clickstack_sql with a connectionId and queries: [{ "id": "a", "sql": "SELECT 1 AS v" }, { "id": "b", "sql": "SELECTT 1" }].
  2. Confirm the response has summary: { total: 2, ok: 1, error: 1, timeout: 0 }, rows for a, and a syntax error for b.
  3. Call it with only sql: "SELECT 1" and confirm the response matches main.

How to test on Vercel preview

N/A — non-UI change.

References

  • Linear Issue: HDX-5747
  • Related PRs:

@vercel

vercel Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Nov 3, 2026 9:32pm UTC
hyperdx-storybook Ready Ready Preview Nov 3, 2026 9:32pm UTC

Request Review

@changeset-bot

changeset-bot Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: e7f6db7

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages
Name Type
@hyperdx/api Patch
@hyperdx/app Patch
@hyperdx/otel-collector Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

};
try {
const prepared = await prepareSqlQuery(
{

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Batch runner and its 60s budget duplicate the queryTiles batch loop

runSqlBatch re-implements the loop in packages/api/src/mcp/tools/dashboards/queryTiles.ts (lines 229-312): a PQueue, a shared deadlineAt, withDeadline around runConfigTile with an abort signal, folding each result into an ok/error entry, and a summary count. SQL_BATCH_DEADLINE_MS = 60_000 also restates BATCH_DEADLINE_MS = 60_000 from queryTiles, so the two tools' budgets now have to be kept in sync by hand. The PR already moved withDeadline into mcp/utils/timeout.ts; move the shared deadline constant there too, or a small generic runBatchUnderDeadline(items, concurrency, work) helper, and have both tools import it.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond


/** Total serialized size budget split evenly across the batch's results. */
const SQL_BATCH_RESPONSE_BUDGET = 100_000;
const SQL_SINGLE_RESPONSE_LIMIT = 50_000;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — SQL_SINGLE_RESPONSE_LIMIT restates trimToolResponse's default

SQL_SINGLE_RESPONSE_LIMIT = 50_000 copies the default maxSize = 50000 in packages/api/src/utils/trimToolResponse.ts:17, which is the limit single-query clickstack_sql actually uses. If that default changes, batch mode silently stops matching. Export the default from trimToolResponse.ts (e.g. DEFAULT_TOOL_RESPONSE_MAX_CHARS) and use it in both places.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 critical or 🟠 major in code this PR adds or changes: fix before asking for review.
  • 🔵 minor in code this PR changes: your call. Fix if small, otherwise reply.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    severity. Reply with one sentence; if it is critical, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

3 finding(s): 🔴 0 critical · 🟠 0 major · 🔵 3 minor

2 posted as inline comment(s) on the changed lines. 1 listed below.

Findings outside the changed lines

1 minor
  • 🔵 packages/api/src/mcp/tools/query/sql.ts:1144 — Single-query mode still reads the source from Mongo twice → The single path creates a createSourceLookup cache and uses it only for prepareSqlQuery. runConfigTile doesn't receive it, so it calls getSource again (helpers.ts:644-646). Store the lookup in a variable and pass it as { lookupSource } to runConfigTile, the same way batch mode does.

Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides.

@greptile-apps

greptile-apps Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; no new actionable issues were found.

Summary

Adds parallel queries to clickstack_sql while keeping single-query calls supported.

  • MCP callers can run up to ten SQL queries in one call.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[clickstack_sql] --> B{sql or queries?}
  B -->|sql| C[Prepare and run one query]
  B -->|queries| D[Choose each connection and source]
  D --> E[Prepare under shared deadline]
  E --> F[Run up to five queries at once]
  F --> G[Collect outcomes in input order]
  G --> H[Trim rows and return batch]
Loading

Reviews (4) · Last reviewed commit: "fix(mcp): don't inherit the top-level so..." · Reviewed by Greptile

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

✅ No critical issues found. The batch runner is careful and unusually well-tested — the withDeadline abort propagates to the ClickHouse client, fitToBudget provably terminates, queued-but-unstarted queries fail fast without touching ClickHouse, and the single-sql path is behavior-preserving. All three prior Greptile comments (source lookups escaping the deadline, batch budget enforcement, execution bypassing the source cache) are resolved and covered by tests.

🟡 P2 -- recommended

  • packages/api/src/mcp/tools/query/sqlBatch.ts:149 -- fitToBudget only shrinks row arrays, so when the oversized payload is all error/timeout entries (no rows), the if (!target) return exit returns a response still above SQL_BATCH_RESPONSE_BUDGET; per-query maxResponseChars never caps error-string size, so the 100k guarantee can be missed for an all-failing batch.
    • Fix: after the row-trim loop, enforce the budget on non-row content too (e.g. truncate oversized error strings or the aggregate) before serializing.
    • correctness, adversarial, reliability, testing
🔵 P3 nitpicks (7)
  • packages/api/src/mcp/tools/query/sqlBatch.ts:179 -- per-query maxResponseChars is enforced by trimToolResponse in compact JSON, but fitToBudget and the final payload use 2-space JSON.stringify(payload, null, 2), so the per-query total (~100k compact) inflates well past the budget pretty-printed and rows get dropped from queries that each respected the stated 50k limit.
    • Fix: measure the per-query budget and fitToBudget in the same serialization format the response is emitted in.
  • packages/api/src/mcp/tools/query/sqlBatch.ts:77 -- SqlBatchEntry's [key: string]: unknown index signature removes compile-time checking of result/note, and { ...base, status, ...parsePayload(...) } spreads parsed payload last, so a future runConfigTile payload key named index/status/id would silently clobber the batch's own fields.
    • Fix: declare result/note/hints explicitly and drop the index signature, or nest the parsed payload under a typed field instead of spreading it onto the entry.
    • maintainability, kieran-typescript
  • packages/api/src/mcp/tools/query/prepareSql.ts:23 -- the source lookup memoizes the getSource promise, so a single transient Mongo rejection is cached and fails every query in the batch that shares that sourceId, even though an independent read might have succeeded.
    • Fix: cache only resolved sources, or evict the cache entry when the lookup rejects.
    • correctness, adversarial, reliability, kieran-typescript
  • packages/api/src/mcp/tools/query/sqlBatch.ts:252 -- the mcp.sql.batch.trimmed span attribute counts every entry with a note, including the per-query trimToolResponse note that formatQueryResult sets, so it overcounts batch-level trims.
    • Fix: track entries actually trimmed by fitToBudget rather than inferring from note.
    • correctness, kieran-typescript, adversarial
  • packages/api/src/mcp/tools/query/sqlBatch.ts:239 -- a non-Error throw falls to String(err) (yielding [object Object] for object throws), and any unexpected err.message is surfaced verbatim to the client, unlike the structured single-query path.
    • Fix: serialize non-Error throws usefully and wrap unexpected internal errors in a generic message.
    • testing, reliability, adversarial
  • packages/api/src/mcp/tools/query/sqlBatch.ts:188 -- prepareSqlQuery (and its getSource lookups) runs via Promise.all outside the PQueue, so up to 10 concurrent Mongo reads can fire despite the advertised concurrency of 5.
    • Fix: note the bound or route source resolution through the queue if Mongo pressure is a concern.
    • adversarial, reliability
  • .changeset/mcp-sql-batch-queries.md:2 -- this additive, backward-compatible feature uses a patch bump while recent feat(mcp) changesets used minor; repo precedent is mixed, so confirm the intended semver.
    • Fix: change to minor if the new queries capability warrants a feature bump.

Reviewers (9): correctness, adversarial, testing, maintainability, kieran-typescript, reliability, performance, previous-comments, project-standards.

Testing gaps:

  • No test for the fitToBudget all-error/no-droppable-rows path (the if (!target) return over-budget exit) or for halving a single oversized row to zero.
  • No test rejects a query with a non-Error value, so the String(err) branch is uncovered.
  • No test covers a getSource rejection being cached and shared across sibling queries with the same sourceId.
  • parsePayload's non-JSON / array { raw: text } fallback is never exercised.
  • The withDeadline abort reaching runConfigTile's ClickHouse abort_signal is not verified end-to-end (mocks ignore the signal).

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

return { raw: text };
}

/** The row array of an ok entry: `result` itself, or ClickHouse's `result.data`. */

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — rowsOf re-implements the row extraction in summarizeRows

summarizeRows in packages/api/src/mcp/tools/dashboards/queryTiles.ts (L89-100) already turns a formatQueryResult payload into rows: result when it is an array, or result.data. Move that into a shared helper (e.g. extractRows(result) next to formatQueryResult in query/helpers.ts) and use it from both places, so a change to the payload shape only has to be made once. The 60s deadline is also duplicated: BATCH_DEADLINE_MS in queryTiles.ts L59 and SQL_BATCH_DEADLINE_MS here. Share it from mcp/utils/timeout.ts, next to withDeadline.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

…-queries

# Conflicts:
#	packages/api/src/mcp/tools/query/sql.ts
@github-actions github-actions Bot added the review/tier-3 Standard — full human review required label Oct 20, 2026
@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

🟡 Tier 3 — Standard

Introduces new logic, modifies core functionality, or touches areas with non-trivial risk.

Why this tier:

  • Diff size: 585 production lines changed (Tier 2 max: < 250)

Review process: Full human review — logic, architecture, edge cases.
SLA: First-pass feedback within 1 business day.

Stats
  • Production files changed: 7
  • Production lines changed: 585 (+ 674 in test files, excluded from tier calculation)
  • Branch: brandon/mcp-sql-batch-queries
  • Author: brandon-pereira

To override this classification, remove the review/tier-3 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Duplicate code analysis

No new duplicate code introduced by this PR.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 114 in total (1.40%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

* ClickHouse's own max_execution_time; this bounds the call as a whole so a
* queue of slow queries cannot run past the client's patience.
*/
export const SQL_BATCH_DEADLINE_MS = 60_000;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Batch deadline constant and batch-runner pattern duplicated from queryTiles.ts

SQL_BATCH_DEADLINE_MS = 60_000 restates BATCH_DEADLINE_MS in packages/api/src/mcp/tools/dashboards/queryTiles.ts:60. Both files also repeat the same PQueue + shared deadlineAt + per-item try/catch + status-summary loop. withDeadline was moved to mcp/utils/timeout.ts, so export one batch deadline constant from there too (ideally with a small shared runner) instead of keeping two copies that must be kept in sync by hand.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 460 passed • 1 skipped • 1561s

Status Count
✅ Passed 460
❌ Failed 0
⚠️ Flaky 2
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

knudtty
knudtty previously approved these changes Nov 2, 2026

@knudtty knudtty left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very good idea

@kodiakhq
kodiakhq Bot requested a review from a team as a code owner November 2, 2026 20:36
@kodiakhq
kodiakhq Bot requested a review from jordan-simonovski November 2, 2026 20:36
}
const payload = { summary, results };
fitToBudget(payload, SQL_BATCH_RESPONSE_BUDGET);
const trimmed = results.filter(entry => entry.note !== undefined).length;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — mcp.sql.batch.trimmed also counts queries that runConfigTile trimmed, not just ones the batch budget trimmed

parsePayload copies runConfigTile's own note ("Result was trimmed for context size…", from formatQueryResult in helpers.ts) onto the entry. So entry.note !== undefined is true for every query trimmed per-query, even when fitToBudget never touched it. Track batch trimming separately: have setRows/fitToBudget return or record the indices it trimmed, and count those.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

}

/** The row array of an ok entry: `result` itself, or ClickHouse's `result.data`. */
function rowsOf(entry: SqlBatchEntry): unknown[] | undefined {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — rowsOf repeats the row extraction in summarizeRows

summarizeRows in packages/api/src/mcp/tools/dashboards/queryTiles.ts (lines 90-101) already turns a result that is either an array or { data: [] } into rows. Move that branch into a shared helper (e.g. extractRows(result) in query/helpers.ts next to formatQueryResult) and use it in both places. The DRY rule in the conventions is marked REQUIRED.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

@kodiakhq

kodiakhq Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

This PR currently has a merge conflict. Please resolve this and then re-add the automerge label.

…-queries

# Conflicts:
#	packages/api/src/mcp/tools/dashboards/queryTiles.ts
}

/** The row array of an ok entry: `result` itself, or ClickHouse's `result.data`. */
function rowsOf(entry: SqlBatchEntry): unknown[] | undefined {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — rowsOf duplicates the row extraction in summarizeRows

summarizeRows in packages/api/src/mcp/tools/dashboards/queryTiles.ts:91-102 already reads rows from a payload's result, whether it is a top-level array or a { data: [...] } object. Move that logic into a shared helper next to formatQueryResult in query/helpers.ts, which produces the payload, and have both callers use it.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

* ClickHouse's own max_execution_time; this bounds the call as a whole so a
* queue of slow queries cannot run past the client's patience.
*/
export const SQL_BATCH_DEADLINE_MS = 60_000;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — The 60s batch deadline is defined twice

SQL_BATCH_DEADLINE_MS = 60_000 repeats BATCH_DEADLINE_MS = 60_000 in packages/api/src/mcp/tools/dashboards/queryTiles.ts:61. The PQueue + withDeadline + error-to-status loop is also repeated (queryTiles.ts:233-301). withDeadline already moved to mcp/utils/timeout.ts, so put one exported batch deadline constant there and have both tools import it.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

@kodiakhq
kodiakhq Bot merged commit d08b2fa into main Nov 3, 2026
29 checks passed
@kodiakhq
kodiakhq Bot deleted the brandon/mcp-sql-batch-queries branch November 3, 2026 22:20

This branch was successfully deployed

2 active deployments
Preview – hyperdx-oss — e7f6db77 Deployed Nov 3, 2026 by vercel[bot]
Preview – hyperdx-storybook — e7f6db77 Deployed Nov 3, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

automerge review/tier-3 Standard — full human review required

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants