Skip to content

fix(mcp): return partial describe_source results on timeout - #3259

Merged
kodiakhq[bot] merged 9 commits into
mainfrom
brandon/mcp-describe-source-timeout
Oct 2, 2026
Merged

kodiakhq[bot] merged 9 commits into
mainfrom
brandon/mcp-describe-source-timeout

Conversation

@brandon-pereira

@brandon-pereira brandon-pereira commented Oct 1, 2026 •

Copy link
Copy Markdown
Member

Why

clickstack_describe_source times out often in production, and successful calls run close to the 10s cap. Our server instructions tell agents to run describe_source before querying, so when it fails they fall back to clickstack_sql.

Two problems:

  • The 10s budget is a third of what the other MCP query tools get (max_execution_time 30, request timeout 32s via MCP_REQUEST_TIMEOUT).
  • On timeout, Promise.race rejected and threw away everything gathered so far, including the column schema. The existing skippedStages partial-result logic never ran.

What changed

  • DESCRIBE_TIMEOUT_MS 10s → 30s.
  • At the deadline the handler aborts the signal. Discovery skips its remaining stages and returns columns plus whatever else finished, with partial: true and skippedStages.
  • A 2s backstop (32s total) covers ClickHouse calls that ignore the signal: it answers from a snapshot of what discovery has gathered so far.
  • The "Schema discovery timed out" error is now only returned when the column schema never loaded.
  • Metric sources now report metricNames in skippedStages when the abort lands before that stage (it was silently omitted).
  • Tool description mentions the 30s limit and the partial-result shape.
  • Span attribute mcp.describe_source.outcome (complete / partial / timeout) so we can track the effect.

Testing

  • New unit tests in describeSourceTimeout.test.ts (fake timers): complete path, abort mid-map-keys, abort mid-value-sampling, backstop when a call ignores the signal, and the column-schema-never-loads error.
  • yarn ci:unit in packages/api: 1080 passed.
  • make dev-int FILE=src/mcp/__tests__/sources.int.test.ts: 55 passed.

🤖 Generated with Claude Code

Raise the clickstack_describe_source deadline from 10s to 30s to match the
other MCP query tools. At the deadline, abort the signal and return whatever
discovery has gathered (columns at minimum, with skippedStages) instead of
discarding it. A 2s backstop answers from the latest snapshot if a ClickHouse
call ignores the signal. The timeout error is only returned when the column
schema never loaded.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@vercel

vercel Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Oct 2, 2026 1:58pm UTC
hyperdx-storybook Ready Ready Preview Oct 2, 2026 1:58pm UTC

Request Review

@changeset-bot

changeset-bot Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: f3e4451

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages
Name Type
@hyperdx/api Patch
@hyperdx/app Patch
@hyperdx/otel-collector Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions github-actions Bot added the review/tier-2 Low risk — AI review + quick human skim label Oct 1, 2026
@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

🔵 Tier 2 — Low Risk

Small, isolated change with no API route or data model modifications.

Why this tier:

  • Standard feature/fix — introduces new logic or modifies core functionality

Review process: AI review + quick human skim (target: 5–15 min). Reviewer validates AI assessment and checks for domain-specific concerns.
SLA: Resolve within 4 business hours.

Stats
  • Production files changed: 1
  • Production lines changed: 248 (+ 356 in test files, excluded from tier calculation)
  • Branch: brandon/mcp-describe-source-timeout
  • Author: brandon-pereira

To override this classification, remove the review/tier-2 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Duplicate code analysis

No new duplicate code introduced by this PR.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 127 in total (1.80%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 critical or 🟠 major in code this PR adds or changes: fix before asking for review.
  • 🔵 minor in code this PR changes: your call. Fix if small, otherwise reply.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    severity. Reply with one sentence; if it is critical, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

3 finding(s): 🔴 0 critical · 🟠 0 major · 🔵 3 minor

2 posted as inline comment(s) on the changed lines. 1 listed below.

Findings outside the changed lines

1 minor
  • 🔵 packages/api/src/mcp/tools/sources/describeSource.ts:660 — Thrown errors never set mcp.describe_source.outcome → The 'error' outcome is only recorded when the result has isError. The rethrow path in the catch block (throw e) skips recordOutcome, so spans for thrown failures have no outcome attribute. That breaks counting outcomes against the total call volume. Call recordOutcome('error') before throw e.

Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides.

@greptile-apps

greptile-apps Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

[Medium risk] Changes how the describe source tool handles timeouts.

The PR appears safe to merge; no new actionable issue was identified.

Summary

The PR extends the describe-source deadline and returns available schema and samples as a marked partial result when discovery runs out of time.

  • A backstop returns a snapshot if discovery does not finish after abort.
  • Tests cover interrupted sampling, backstop responses, and metric-source results.
Diagram
%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Describe source] --> B[Load column schema]
  B --> C[Sample optional metadata]
  C -->|Finished| D[Complete result]
  C -->|30s deadline| E[Abort sampling]
  E -->|Settles| F[Partial result with skipped stages]
  E -->|Still pending at 32s| G[Snapshot]
  G -->|Schema loaded| F
  G -->|No schema| H[Timeout error]
Loading

Reviews (9) · Last reviewed commit: "Merge branch 'main' into brandon/mcp-des..."

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

✅ No critical issues found.

This is a focused, well-tested change to the clickstack_describe_source timeout path. I verified the central reliability claim directly: there is no server- or transport-level timeout on the MCP handler (mcpServer.ts/app.ts impose none) and describeSource constructs its own ClickhouseClient with no requestTimeout, so the 32s backstop is the effective wall-clock cap and the partial-result path is genuinely reachable in production. The backstop snapshot is safe against the concurrent-mutation concern because the shallow copy is serialized by a synchronous JSON.stringify before any pending map callback can interleave. Losing-branch rejections after the backstop fires are consumed by Promise.race (and internal stages swallow their own rejections), so there is no unhandled-rejection path. Both prior review rounds (the greptile P1 on interrupted-sampling accounting and P2 on missing metric-source tests) are addressed in the current diff.

🟡 P2 — recommended

  • packages/api/src/mcp/tools/sources/describeSource.ts:460 — the new mcp.describe_source.outcome span attribute (the observability signal the change adds to measure its own effect) is set on every exit path but no test asserts it, so a regression that silently stops emitting partial/timeout/complete would go unnoticed.
    • Fix: add an assertion on the recorded span attribute (via a mocked active span) in the complete, partial, and column-schema-never-loads cases.
🔵 P3 nitpicks (1)
  • packages/api/src/mcp/tools/sources/describeSource.ts:256 — stage-completion state is tracked three ways (skippedStages inline pushes, finishedStages, and the optionalStages reconstruction in the snapshot closure) with per-stage conditions that must stay mutually consistent; adding or reordering a discovery stage could desync the backstop snapshot from the inline accounting without an obvious test failure.
    • Fix: consider centralizing per-stage status in a single structure (e.g. one map of stage → pending|finished|skipped) so the snapshot and final result derive skippedStages from one source of truth.

Reviewers (2): orchestrator deep-analysis (correctness, reliability, maintainability, and testing lenses applied directly to the diff), previous-comments.

Testing gaps:

  • The outcome span attribute is emitted but never asserted.
  • No test exercises discovery resolving in the same tick the deadline fires (Promise.race tie); low risk, covered indirectly by fake-timer ordering.

Residual risk (advisory, intentional): raising the deadline 10s→30s holds ClickHouse discovery queries up to ~3x longer per call, which can increase concurrent query pressure under load — a deliberate trade-off to match the other MCP query tools' budget.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 393 passed • 1 skipped • 1179s

Status Count
✅ Passed 393
❌ Failed 0
⚠️ Flaky 1
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

function getHandler() {
let handler: ((args: { sourceId: string }) => Promise<ToolResult>) | null =
null;
registerDescribeSource({

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — New metric-source behaviour and the outcome span attribute have no tests

Every test uses a Trace source, so nothing checks that metric sources now list metricNames in skippedStages, either in the abort-before-stage-5 path or in the backstop snapshot (via optionalStages). That was one of the behaviours this PR changed on purpose. Nothing checks mcp.describe_source.outcome either. Fix: add a SourceKind.Metric fixture whose metricTables are populated, abort during map keys, and assert skippedStages contains metricNames. Also add a test that mocks trace.getActiveSpan and asserts the outcome attribute.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

const DESCRIBE_TIMEOUT_MS = 10_000;
// When discovery's signal is aborted; it then returns what it has so far.
// Matches the 30s budget of the other MCP query tools.
export const DESCRIBE_TIMEOUT_MS = 30_000;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — The 30s/32s budget is copied by hand from query/helpers.ts

DESCRIBE_TIMEOUT_MS = 30_000 and DESCRIBE_BACKSTOP_MS = 2_000 repeat max_execution_time: 30 and MCP_REQUEST_TIMEOUT = 32_000 from packages/api/src/mcp/tools/query/helpers.ts:75,84. They stay in sync only because a comment says so. Fix: export MCP_REQUEST_TIMEOUT (and the 30s limit) from helpers.ts and derive both constants from them.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

Report mapAttributeKeys / metricNames as skipped when the deadline hits
after only some Map columns or metric kinds finished, keep finished
results in the backstop snapshot, leave stages with nothing to sample out
of it, and derive the outcome span attribute from the returned result.
Add metric-source and partial-stage tests.

expect(source.partial).toBe(true);
expect(source.skippedStages).toEqual(['metricNames']);
});

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Tests mock abort as a rejection, but the real helpers resolve empty on abort

rejectOnAbort makes getAllKeyValues and sampleMetricNamesWithLookback reject when aborted. The real Metadata.getAllKeyValues (common-utils/src/core/metadata.ts:2748-2761) swallows the abort and resolves [], and so does the lookback helper. So the skip-detection paths are tested against behaviour production never shows, which is why the metricNames bug above passes. Mock these to resolve [] on abort, matching the real helpers, and assert skippedStages.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

'Schema discovery timed out. The ClickHouse server may be under load. ' +
'Try again, or use clickstack_list_sources for basic source info without schema details.',
);
if (controller.signal.aborted) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Any error thrown after the 30s abort is reported as a timeout, hiding the real cause

The catch block returns timedOutResult() whenever controller.signal.aborted is true. It doesn't check whether the error was actually an abort. getColumns, getSource and getConnectionById take no signal. If one of them throws a real error between 30s and 32s (for example a missing table or a Mongo failure), the agent gets 'Schema discovery timed out before the column schema loaded' and is told to retry, and the real error is never logged. Only use the timeout result for abort errors (check e.name === 'AbortError', or check that the error came from a signalled call). Otherwise log and rethrow as the pre-deadline path does.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

…w major)

getAllKeyValues drops aborted chunks instead of throwing, so a stage that
returns some values after the abort is still incomplete. Also report
mapAttributeValues as skipped when map keys were cut short, matching the
backstop snapshot.
clearTimeout(abortTimer);
clearTimeout(backstopTimer);
}
},

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — After the abort, any error is reported as a timeout and never logged

In the catch block, controller.signal.aborted sends every rejection to timedOutResult() before the logger.warn. So a real failure that lands after 30s is returned as "timed out before the column schema loaded" with outcome timeout, and the cause is dropped. Examples: getColumns hitting UNKNOWN_TABLE (it takes no signal), or a Mongo error from getConnectionById. Log e before returning, and only use the timeout path when the error is actually an abort (e.g. e.name === 'AbortError'). Rethrow anything else.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond


expect(result.isError).toBeFalsy();
const { source, usage } = parse(result);
expect(source.columns).toHaveLength(COLUMNS.length);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — The new outcome span attribute has no test

The PR adds mcp.describe_source.outcome (complete/partial/timeout) as the way to measure the fix, but no test mocks @opentelemetry/api's trace.getActiveSpan. A wrong or missing outcome would still pass. Mock getActiveSpan the way src/mcp/__tests__/tracing.test.ts does, and assert the attribute value in the complete, abort-partial, backstop-partial and schema-timeout cases.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

@kodiakhq
kodiakhq Bot merged commit c2e21cf into main Oct 2, 2026
29 checks passed
@kodiakhq
kodiakhq Bot deleted the brandon/mcp-describe-source-timeout branch October 2, 2026 14:02

This branch was successfully deployed

2 active deployments
Preview – hyperdx-oss — f3e44519 Deployed Oct 2, 2026 by vercel[bot]
Preview – hyperdx-storybook — f3e44519 Deployed Oct 2, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

automerge review/tier-2 Low risk — AI review + quick human skim

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants