Skip to content

feat(mcp): add clickstack_service_health tool - #3411

Merged
kodiakhq[bot] merged 17 commits into
mainfrom
brandon/brandon-mcp-service-health-tool
Oct 21, 2026
Merged

kodiakhq[bot] merged 17 commits into
mainfrom
brandon/brandon-mcp-service-health-tool

Conversation

@brandon-pereira

Copy link
Copy Markdown
Member

Summary

Adds clickstack_service_health, an MCP tool that answers "how are my services doing?" in one call. Today an agent stitches this together from several clickstack_table / clickstack_timeseries calls and then has to judge on its own which changes matter.

For every service on a trace source, the tool returns throughput, error rate, p50/p95/p99 and the top error groups. Each value is compared with the same window yesterday, and services come back ranked worst first. Each service gets:

  • a status (critical / degraded / healthy) and the flags behind it (error_rate_up, latency_up, traffic_drop, ...). Flags fire only past fixed thresholds, so routine variance is not reported as a regression.
  • a one-line summary of which changes crossed a threshold.
  • error groups that are new or growing versus the comparison window, listed first, with first/last-seen times so a burst that already stopped reads as transient.
  • operations with no traffic in the comparison window, listed as likely rollouts. Errors on those operations are not counted as new errors.

Behavior worth knowing:

  • Counts server spans by default (spanKind: "all" to include every span).
  • If the comparison window has no data (for example, beyond retention), the tool falls back to the previous period, and to no comparison if that is empty too, so nothing is falsely labelled new.
  • Error novelty is checked across all services before cutting to topN, so a quiet service with a brand-new failure still surfaces.

The span error predicate moves from packages/app/src/source.ts to packages/common-utils/src/core/spanPredicates.ts (with a server span-kind predicate) so the API and the app share one definition. No behavior change in the app.

Eval results

A/B against main with packages/hdx-eval, both arms on the same base commit, 5 runs per arm.

Scenario This branch main Δ 95% CI
service-health-check 70.6% 63.7% +7.0 +0.1 to +13.8
error-root-cause 98.5% 94.1% +4.5 +1.5 to +7.5
  • service-health-check is the use case this tool is for: a routine status report where the agent has to report SLIs, call out what is new, and not escalate routine variance. Agents also used about half the tool calls.
  • error-root-cause: in every branch run, the tool's top result was the service with the new failure (checkout-api, a new payment_service_unavailable error group), which points the agent straight at the root cause.

How to reproduce

Starting point: local full stack (yarn dev) with some trace data, and an MCP client (for example npx @modelcontextprotocol/inspector) connected to the local MCP endpoint (see MCP.md).

On main:

  1. Run tools/list. There is no clickstack_service_health.

On this branch:

  1. Run tools/list. clickstack_service_health is listed.
  2. Call clickstack_list_sources and copy the id of a kind: "trace" source.
  3. Call clickstack_service_health with { "sourceId": "<id>" }.
  4. The response lists services worst first, each with status, flags, summary, current and comparison metrics, and topErrors.
  5. Call it with a table name as sourceId (for example "otel_traces"). The error names the correct source ID.

To run the evals: packages/hdx-eval README, scenarios service-health-check and error-root-cause.

How to test on Vercel preview

N/A — non-UI change.

References

  • Linear Issue: HDX-5745

One-call per-service health for a trace source: throughput, error rate,
p50/p95/p99 and top error groups, compared with yesterday (falling back to
the previous period when that window is empty) and ranked worst first.
New and growing error groups are listed first with first/last-seen times,
and operations with no prior traffic are reported as likely rollouts.

Moves the span error/server-kind predicates to common-utils so the API and
app share one definition.
… window boundaries

- Assess error novelty for up to 100 services before cutting to topN. A
  service ranked below topN by aggregate score was never checked for new
  errors, so a new failure on a quiet service could not surface.
- Make every window half-open. With an inclusive end, a span at exactly the
  boundary counted in both windows, so an empty comparison could look like a
  one-request baseline.
- Treat fewer than minRequests prior requests as no baseline (new_service),
  and do not call that service's errors or operations new.
- Exclude errors on a brand-new operation from new_errors and the summary.
  This was intended earlier but the change had not applied.
- Drop "no action needed" from a healthy summary that lists new errors.
@vercel

vercel Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Oct 20, 2026 7:10pm UTC
hyperdx-storybook Ready Ready Preview Oct 20, 2026 7:10pm UTC

Request Review

@changeset-bot

changeset-bot Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: b11c5c3

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages
Name Type
@hyperdx/api Minor
@hyperdx/common-utils Patch
@hyperdx/app Minor
@hyperdx/otel-collector Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Duplicate code analysis

No new duplicate code introduced by this PR.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 123 in total (1.61%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

@greptile-apps

greptile-apps Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 4/5

[Medium impact] The main effects stay within the MCP trace-service health workflow.

Fix the error timestamp selection before merging; active failures can otherwise be reported as finished.

Fix All in Claude CodeFindings

  1. P1 Active errors look finished ▶
  2. P2 Rollouts look like new failures ▶

Summary

Adds clickstack_service_health for ranked service metrics, comparison windows, error groups, and likely rollouts.

  • The MCP service health tool ranks trace services by their current health.
  • New and growing errors stand out from routine service traffic.
  • The app and API use shared rules to identify error and server spans.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  Input[Service health request] --> Source[Load team trace source]
  Source --> Stats[Fetch current and comparison metrics]
  Stats --> Fallback[Choose available comparison window]
  Fallback --> Rank[Assess and rank services]
  Rank --> Errors[Compare error groups and find new operations]
  Errors --> Report[Return ranked report and coverage notes]
Loading

Reviews (13) · Last reviewed commit: "Merge branch 'main' into brandon/brandon..." · Reviewed by Greptile

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 456 passed • 1 skipped • 1521s

Status Count
✅ Passed 456
❌ Failed 0
⚠️ Flaky 1
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

…-clean

- Report a coverageNote when a limit (services checked, error rows, groups
  per service) kept errors out of the novelty check (greptile P1).
- Skip new-operation detection when either operations query is truncated,
  so an existing operation cannot look new (greptile P1).
- Classify rollout errors against every new operation; cap only the
  returned list (greptile P2).
- Omit new/growing labels on errors of services without a baseline
  (greptile P2).
- Read result rows through a Map and type the test's parsed body, keeping
  the api lint warning count at its cap (lint check).
@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

🔴 Tier 4 — Critical

Touches authentication, tenancy data models, the public API or shipped database config — or substantially changes the query rendering engine, background tasks, the OTel pipeline, image build, or release CI.

Why this tier:

  • Large diff: 1706 production lines changed (threshold: 1000)
  • Cross-layer change: touches frontend (packages/app) + backend (packages/api) + shared utils (packages/common-utils)

Review process: Deep review from a domain expert. Synchronous walkthrough may be required.
SLA: Schedule synchronous review within 2 business days.

Stats
  • Production files changed: 17
  • Production lines changed: 1706 (+ 839 in test files, excluded from tier calculation)
  • Branch: brandon/brandon-mcp-service-health-tool
  • Author: brandon-pereira

To override this classification, remove the review/tier-4 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

if (fallback && fallbackCmp?.stats.size) {
return {
comparison: fallback,
cmp: fallbackCmp,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Empty filtered comparison is treated as a retention gap, so a genuinely new service loses its new_service flag

requestedCmp comes from baseConfig, which already applies input.services, where and spanKind. So stats.size === 0 also happens when the filter only matches new traffic, for example services:["svc-launched-today"]. In that case the tool falls back to previous_period and then to none, blames retention in comparisonNote, and never reports the new_service flag. Fix: decide the fallback with an unfiltered existence probe on the comparison window (time range only), and keep the requested comparison when the table does have data there.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

const { summary, services } = parse(await call());
expect(services.map(s => [s.service, s.status])).toEqual([
[svc('erroring'), 'critical'],
[svc('slow'), 'degraded'],

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Integration test never runs the comparison fallback, new-operation detection, or the novelty lift against real ClickHouse

Every seeded span is named 'GET /api' and both windows have data, so findNewOperations always returns empty and resolveComparison always returns the requested window. A regression that broke the operations SQL, or made the fallback never fire, would still pass. The unit tests stub these inputs (newOperations: new Map()). Fix: add an int case that seeds a span name present only in the current window and checks newOperations plus status:'new_operation'. Add another case with no yesterday data that checks summary.comparison === 'previous_period' and comparisonNote.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 critical or 🟠 major in code this PR adds or changes: fix before asking for review.
  • 🔵 minor in code this PR changes: your call. Fix if small, otherwise reply.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    severity. Reply with one sentence; if it is critical, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

4 finding(s): 🔴 0 critical · 🟠 2 major · 🔵 2 minor

None could be anchored to changed lines; they are listed below.

Findings outside the changed lines

  • 🟠 packages/api/src/mcp/tools/query/serviceHealth/summary.ts:498 — Ingestion lag makes an ongoing error read as a stopped burst that is 'transient, not an ongoing problem' → endTime defaults to now, and ongoing only holds if the last error is within 2 minutes of window.end. Take a span pipeline that runs 3 minutes behind (batch exporters often do) and a failure that started at 17:55 and is still firing, with the window ending at 18:05. The last ingested error is at 18:02. It is not ongoing, and its spread is 7/60, so it is classed burst. The summary then tells the agent it is transient and can be ignored. Measure the tail against the newest span actually seen in the window (for example max(timestamp) per service from fetchServiceStats) instead of window.end. Or skip the 'transient' wording when window.end is within a few minutes of now.
  • 🟠 packages/api/src/mcp/tools/query/serviceHealth/errorGroups.ts:1206 — Full status messages (often stack traces) go into the report twice and push services out of the response → Group values are returned at full length. describeError (summary.ts:517) then copies the same label into every service's summary. trimToolResponse (packages/api/src/utils/trimToolResponse.ts:98) splits its 50KB budget evenly across the three top-level keys. That leaves services about 16.6KB, so a handful of multi-KB exception messages cuts the list to 10 services. summary.servicesReturned still reports the untrimmed count. Cap each group value (for example left(toString(e), 200) in the query, or truncate in JS) and keep the summary label short.
2 minor
  • 🔵 packages/api/src/mcp/tools/query/serviceHealth/summary.ts:2568 — A low-volume service is told to be reported as 'steady, no action needed' whatever its error rate → A low_volume service has no degrading flags, so the head becomes 'Healthy: no metric crossed a threshold … report this service as steady, no action needed.' That happens even at 9 errors in 10 requests (the case in serviceHealthRank.test.ts). The context then adds 'Too few requests to judge.', which contradicts it. Give low_volume its own head (for example 'Not assessed: under minRequests in both windows') instead of the steady verdict.
  • 🔵 packages/api/src/mcp/tools/query/serviceHealth/source.ts:388 — The 'table name passed as sourceId' fix only helps this one tool → Every other source-taking MCP tool still returns the bare 'Source not found: … Call clickstack_list_sources' message (query/helpers.ts:476, query/sql.ts:126, trace/waterfall.ts:349, trace/breakdown.ts:84, query/eventDeltas.ts:245, query/runEventPatterns.ts:71). Agents make the same mistake with those tools. Move sourceNotFound into a shared MCP helper, parameterized by source kind, and use it from those call sites too.

Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides.

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

✅ No critical issues found. This is a carefully written, well-decomposed addition. The five substantive prior-review findings (Greptile) were verified as addressed in the current code: collect.ts now emits an explicit coverageNote when limits truncate the new-error check; operations.ts propagates a truncated/complete flag so a globally-limited comparison no longer mislabels existing operations as rollouts; new-operation error classification uses the full set (display limit applied only to output); report.ts gates topErrors novelty labels behind hasBaseline(r); and queries.ts is back under the 300-line guideline (192 lines). SQL construction escapes service names via escapeSqlString/sqlList, cross-window error groups are matched by a validated numeric cityHash64, and window math uses half-open ranges to avoid double-counting at the boundary.

🟡 P2 -- recommended

  • packages/api/src/mcp/tools/query/serviceHealth/index.ts:105 -- The AbortController is only ever .abort()-ed inside the catch block, after the awaited queries have already settled, and its signal is not wired to any upstream MCP request-cancellation, so a client that disconnects mid-call leaves the ClickHouse aggregations running to completion.
    • Fix: Wire the tool invocation's cancellation signal into ctx.signal, or remove the AbortController if no cancellation source is available.
🔵 P3 nitpicks (2)
  • packages/api/src/mcp/tools/query/serviceHealth/report.ts:205 -- "Is this error group newly notable" is computed in two places: selectTopErrors's novel predicate (consumed by collect.ts) and buildReport's changedErrors, each re-deriving the new/growing + explained/new-operation gate; they agree today but can drift as thresholds change.
    • Fix: Extract a single shared isNotableError(group, comparisonCounts, isNewOperation) predicate and call it from both.
  • packages/api/src/mcp/tools/query/serviceHealth/collect.ts:105 -- The MAX_ERROR_SERVICES (100) cap means services ranked beyond 100 never get a new-error check, which partially undercuts the "a quiet service with a brand-new failure still surfaces" goal; this is disclosed via coverageNote but only as free text.
    • Fix: Consider surfacing the cap as a structured field (e.g. servicesCheckedForErrors) alongside the prose note so agents can reason about completeness programmatically.

Reviewers (2): previous-comments, orchestrator deep-read (correctness/security/reliability/performance file-by-file).

Coverage note: The parallel fan-out (correctness, adversarial, performance, reliability, kieran-typescript, testing, maintainability, project-standards) did not return results within this run's window; findings above come from the completed previous-comments reviewer and the orchestrator's own verification. A re-run is advisable if full multi-persona corroboration is required before merge.

Testing gaps: Dedicated unit coverage appears thin for a few pure functions worth pinning down — errorGroupStatus (z-score growing/new boundary), selectTopErrors ordering with the explained callback, findNewOperations truncation handling, and errorTiming pattern classification (ongoing/intermittent/burst); rankServices/resolveComparison and the predicate helpers do have tests.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

groups,
comparisonErrorCounts,
perService,
g => !baseline.get(service) || newOps.has(groupValue(g, spanKey) ?? ''),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — The rule for a new or growing error group is written twice and the copies must be kept in sync by hand

collect.ts passes g => !baseline.get(service) || newOps.has(groupValue(g, spanKey)) to selectTopErrors. report.ts:204 (changedErrors) rebuilds the same rule as hasBaseline(r) && !onNewOperation(r, g) plus isChanged. If one copy changes and the other does not, the groups listed first will disagree with the new_errors flag and the summary. Fix: export one isNovelError(r, g, comparisonCounts, newOperations, spanKey) from report.ts and use it in both places, or have collectErrors return the changed groups for buildReport to reuse.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

* Agents often pass a table or source name where the ID belongs. Point them at
* the exact ID when the input names one trace source, else list the options.
*/
async function sourceNotFound(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — The hint that maps a name or table to its source ID only helps this one tool

sourceNotFound covers a mistake agents make on any tool that takes a sourceId. Every other tool still returns a bare 'Source not found: … Call clickstack_list_sources' (helpers.ts:476, sql.ts:126, eventDeltas.ts:245, trace/waterfall.ts:349, trace/breakdown.ts:84, runEventPatterns.ts:71). Fix: move it into a shared helper such as mcp/tools/query/helpers.ts with a kind parameter, and call it from those not-found branches.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

cmp: null,
comparisonMode: 'none',
comparisonNote:
`The "${requestedMode}" window${fallback ? ' and the previous period have' : ' has'} ` +

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — The empty-window check uses the user's where/services filters, so a filter that only matches new data looks like a retention gap

Scenario: a post-deploy check with where: "ServiceVersion = 'v2'" or services: ["<service launched 40m ago>"]. requestedCmp is empty only because the filter matches nothing yesterday. The tool then falls back to previous_period, and if that is empty too, sets comparisonMode: 'none'. So the service loses new_service, its error groups are never marked new, and the note blames retention. Fix: run the 'does the comparison window have data' probe without input.where/input.services (for example a count() with only the time and spanKind filters), and fall back only when the unfiltered window is empty.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

✅ No critical issues found. This is a carefully decomposed, well-tested addition. Eleven reviewers ran; the core ranking/scoring math, novelty detection, comparison-window fallback, cross-window cityHash64 matching, and half-open window arithmetic were traced and hold up (divide-by-zero and NaN are guarded throughout). The errorStatusPredicateSql / serverSpanKindPredicateSql extraction to common-utils is behavior-preserving and byte-identical, with every in-repo importer repointed and no stale @/source references left. SQL is built with team-scoped, read-only queries following the established MCP pattern (sqlList/escapeSqlString for service names, numeric-only hash filtering, errorGroupBy/where raw-SQL consistent with sibling tools and clickstack_sql); tenant isolation via teamId holds. Of the six substantive prior-review findings, five are verified addressed in the current code; only the AbortController item remains open.

🟡 P2 -- recommended

  • packages/api/src/mcp/tools/query/serviceHealth/errorGroups.ts:104 -- The error-group query applies one global LIMIT 2500 ordered by count descending across all services, so a single high-cardinality service (the default group-by includes statusMessage, often a near-unique stack trace) can consume the entire budget and leave other services with zero error rows, hiding a genuinely new failure while they still report as healthy.
    • Fix: Partition the row budget per service (e.g. a row_number()-bounded per-service cap) instead of a single global limit, so quiet services retain their error slots.
  • packages/api/src/mcp/tools/query/serviceHealth/collect.ts:104 -- The coverageNote limit-assembly branches (partial-services, row-limit, per-service-cap, incomplete-operations) are the mechanism that makes the truncation tradeoffs acceptable, yet collectErrors is imported by no test and the integration fixtures never trip a limit, so a wrong or absent incompleteness note would ship unnoticed.
    • Fix: Add a unit test that drives collectErrors past the truncation and partial-services limits and asserts the assembled coverageNote text.
🔵 P3 nitpicks (6)
  • packages/api/src/mcp/tools/query/serviceHealth/index.ts:105 -- The AbortController is only .abort()-ed in the catch (where it still usefully cancels sibling in-flight queries), but its signal is never wired to upstream MCP request cancellation, so a client that disconnects mid-call leaves the ClickHouse aggregations running until requestTimeout/max_execution_time.
    • Fix: Thread the tool invocation's cancellation signal into ctx.signal if the framework exposes one, or drop the controller if no upstream source is reachable from this layer.
    • previous-comments, reliability
  • packages/api/src/mcp/tools/query/serviceHealth/report.ts:204 -- The "new or growing, not explained by a new operation, with a baseline" gate is derived twice -- once as selectTopErrors' novel predicate (fed the explained closure from collect.ts) and once as buildReport's changedErrors -- so the two can silently diverge, letting topErrors ordering and the new_errors flag disagree.
    • Fix: Extract a single isNovelError(group, prior, { hasBaseline, isNewOperation }) predicate and call it from both sites.
    • maintainability, kieran-typescript
  • packages/api/src/mcp/tools/query/serviceHealth/collect.ts:96 -- When a caller supplies errorGroupBy that omits the source spanNameExpression, groupValue(g, spanKey) is always undefined, so new-operation suppression never matches and errors on brand-new operations are mislabeled new/growing.
    • Fix: Key newOperations on the actual grouped span-name column, or document that new-operation suppression requires the span name in the error grouping.
  • .changeset/mcp-service-health-tool.md:3 -- @hyperdx/common-utils is bumped patch, but the diff adds a new public module (spanPredicates.ts) with two exported functions, which is an additive-minor change under semver.
    • Fix: Change the @hyperdx/common-utils bump to minor (debatable given repo precedent for predicate consolidation and internal-only consumption).
    • project-standards, api-contract
  • packages/api/src/mcp/tools/query/serviceHealth/errorGroups.ts:37 -- groupValue builds new Map(Object.entries(g.group)) to read a single key from what is already a Record<string, string>, allocating a Map per call on a per-error-group path.
    • Fix: Replace with a direct g.group[key] lookup.
  • packages/api/src/mcp/tools/query/serviceHealth/operations.ts:79 -- The priorTotal < minPriorRequests skip (which suppresses per-operation rollout noise for a whole-new service) is exercised by no test; both findNewOperations cases use ample prior traffic.
    • Fix: Add a case where the comparison window has little/no traffic for the service and assert no new operations are reported.
    • testing, correctness

Reviewers (11): correctness, adversarial, security, performance, reliability, kieran-typescript, testing, maintainability, project-standards, api-contract, previous-comments.

Testing gaps: No coverage for the comparison-window error fetch truncating at the 2500-row boundary (bounded by shown≤100 × cap≥25, so edge-only) or cityHash64 collisions; no test for a custom errorGroupBy that excludes the span name and its effect on new-operation suppression; resolveComparison's fallback-to-none note wording is not asserted.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

No trace source we checked has MVs, the error-group queries cannot use one,
and the MV rewrite rounds windows to the view's granularity, which broke the
half-open comparison windows (review minors).
@kodiakhq
kodiakhq Bot merged commit eb72cfa into main Oct 21, 2026
26 checks passed
@kodiakhq
kodiakhq Bot deleted the brandon/brandon-mcp-service-health-tool branch October 21, 2026 15:49
// Match groups across windows by a fixed-size hash. Inlining the raw key
// text (status messages can be stack traces) can exceed max_query_size.
const hashExpr = `cityHash64(${keyExprs.map(e => `toString(${e})`).join(', ')})`;
const epochMs = `toUnixTimestamp64Milli(toDateTime64(${getFirstTimestampValueExpression(source.timestampValueExpression)}, 3))`;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Active errors look finished

With a supported timestamp expression such as EventDate, EventTime, fetchErrorGroups uses only EventDate for the first and last error times. Every error that day then appears at midnight. Even errors still arriving at the end of an afternoon report become a stopped burst, and the summary calls them transient.

Use the highest-precision timestamp column, such as the one selected by pickBucketTimestampColumn, for these times.

Knowledge Base Used: Limit trace breakdown queries and explain timeouts

Fix in Claude Code Fix in Conductor Fix in Cursor Fix in Codex

Comment on lines +189 to +192
const onNewOperation = (r: RankedService, g: ErrorGroup) =>
(args.newOperations.get(r.service) ?? []).some(
o => o.operation === groupValue(g, args.spanNameKey),
);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Rollouts look like new failures

A supported custom grouping such as errorGroupBy: "StatusMessage" removes the span-name key that onNewOperation needs. Errors entirely from a new operation then receive status: "new" instead of new_operation, add new_errors, and move the service ahead in the results. These false warnings can crowd more useful services out of the report.

Keep the operation information needed for this check separately from the requested display grouping.

Fix in Claude Code Fix in Conductor Fix in Cursor Fix in Codex

kodiakhq Bot pushed a commit that referenced this pull request Nov 12, 2026
## Summary

`clickstack_service_health` no longer tells the agent whether a service is critical, degraded or healthy. A healthy service used to get "report this service as steady, no action needed". That verdict comes from service-wide totals, so a regression confined to one endpoint, pod or attribute segment can sit under every threshold while the tool tells the agent to stop looking. In the `deploy-regression` eval, main calls the service with the bug "Healthy".

Each service now carries only evidence:

| | Before | After |
|---|---|---|
| Per-service field | `status: critical / degraded / healthy` | removed; `flags` and `score` stay |
| Summary, nothing crossed | `Healthy: … normal variance; report this service as steady, no action needed.` | `No metric crossed a threshold vs yesterday: error rate 0.50% vs 0.49%, p95 120ms vs 115ms, requests 3000 vs 2900.` |
| Summary, threshold crossed | `CRITICAL: error rate 8.00% vs 0.50%.` | `Crossed thresholds: error rate 8.00% vs 0.50%.` |
| Report summary | `statusCounts: {critical, degraded, healthy}` | `servicesFlagged` |
| Ordering | status, then new errors, then score | threshold-flagged services, then the rest; within each, new or growing errors first, then score |
| Span attributes | `mcp.service_health.critical_count`, `degraded_count` | `mcp.service_health.flagged_count` |

The tool description and `usage` now say that thresholds apply to service-wide totals and point to `clickstack_table` breakdowns before a service is ruled out. "No flags is a valid result for a routine check" stays, so status reports are not pushed into escalating.

`statusCounts` and per-service `status` are gone from the response. Both are MCP output read by agents; nothing else in the repo consumes them.

## Evals

A/B with `packages/hdx-eval`: branch (slot 99) vs `main` (slot 98), both built from `02f6fd060`, identical seeded data, `claude-opus-4-6`.

| Scenario | Runs/arm | Branch | Main | Δ | Tool calls (branch / main) |
|---|---|---|---|---|---|
| `service-health-check` | 3 | 76% | 75% | +1 | 7.7 / 5.7 |
| `error-root-cause` | 3 | 99% | 98% | +1 | 10.3 / 9.0 |
| `segmented-regression` | 8 | 75.5% | 71.6% | +3.9 | 14.9 / 17.1 |
| `deploy-regression` | 8 | 70.3% | 74.6% | −4.3 | 15.6 / 13.9 |
| `quiet-saturation` | 8 | 84.3% | 88.8% | −4.5 | 19.1 / 16.8 |
| **Average** | | **81.0%** | **81.6%** | **−0.6** | |

The change is neutral on scores. Per-run spread within an arm is 15–20 points, so no difference is significant (the largest has t ≈ 1); `segmented-regression` went from −11 after 3 runs to +13 over the next 5. Routine checks did not get more alarmist: the judge scored `service-health-check` 8 points higher on the branch. Its regex false-alarm checks scored it lower, but they mostly match negated phrases such as "not ongoing" and "not escalation-worthy". Agents make a few more tool calls in most scenarios once the tool stops saying "no action needed".

### How to reproduce

**Starting point:** local full stack (`yarn dev`) with trace data, and an MCP client such as `npx @modelcontextprotocol/inspector` connected to the local MCP endpoint (see `MCP.md`).

**On `main`:**

1. Call `clickstack_list_sources` and copy a `kind: "trace"` source ID.
2. Call `clickstack_service_health` with `{ "sourceId": "<id>" }`.
3. Each service has a `status`, and steady services read "Healthy: … no action needed."

**On this branch:**

1. Repeat the call.
2. There is no `status` field. Summaries state the crossed thresholds or the current vs comparison values, and `summary.servicesFlagged` replaces `statusCounts`.

### How to test on Vercel preview

N/A — non-UI change.

### References

- Linear Issue: HDX-5877
- Related PRs: #3411 (added `clickstack_service_health`)

This branch was successfully deployed

2 active deployments
Preview – hyperdx-oss — a8783c7f Deployed Oct 21, 2026 by vercel[bot]
Preview – hyperdx-storybook — a8783c7f Deployed Oct 21, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

automerge review/tier-4 Critical — deep review + domain expert sign-off

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants