Repository navigation
feat(mcp): add clickstack_service_health tool - #3411
Conversation
One-call per-service health for a trace source: throughput, error rate, p50/p95/p99 and top error groups, compared with yesterday (falling back to the previous period when that window is empty) and ranked worst first. New and growing error groups are listed first with first/last-seen times, and operations with no prior traffic are reported as likely rollouts. Moves the span error/server-kind predicates to common-utils so the API and app share one definition.
…eclare all behavior hints
… window boundaries - Assess error novelty for up to 100 services before cutting to topN. A service ranked below topN by aggregate score was never checked for new errors, so a new failure on a quiet service could not surface. - Make every window half-open. With an inclusive end, a span at exactly the boundary counted in both windows, so an empty comparison could look like a one-request baseline. - Treat fewer than minRequests prior requests as no baseline (new_service), and do not call that service's errors or operations new. - Exclude errors on a brand-new operation from new_errors and the summary. This was intended earlier but the change had not applied. - Drop "no action needed" from a healthy summary that lists new errors.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
🦋 Changeset detectedLatest commit: b11c5c3 The changes in this PR will be included in the next version bump. This PR includes changesets to release 4 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Duplicate code analysisNo new duplicate code introduced by this PR. jscpd compared this branch against |
[Medium impact] The main effects stay within the MCP trace-service health workflow. Fix the error timestamp selection before merging; active failures can otherwise be reported as finished.
|
E2E Test Results✅ All tests passed • 456 passed • 1 skipped • 1521s
Tests ran across 4 shards in parallel. |
…-clean - Report a coverageNote when a limit (services checked, error rows, groups per service) kept errors out of the novelty check (greptile P1). - Skip new-operation detection when either operations query is truncated, so an existing operation cannot look new (greptile P1). - Classify rollout errors against every new operation; cap only the returned list (greptile P2). - Omit new/growing labels on errors of services without a baseline (greptile P2). - Read result rows through a Map and type the test's parsed body, keeping the api lint warning count at its cap (lint check).
🔴 Tier 4 — CriticalTouches authentication, tenancy data models, the public API or shipped database config — or substantially changes the query rendering engine, background tasks, the OTel pipeline, image build, or release CI. Why this tier:
Review process: Deep review from a domain expert. Synchronous walkthrough may be required. Stats
|
| if (fallback && fallbackCmp?.stats.size) { | ||
| return { | ||
| comparison: fallback, | ||
| cmp: fallbackCmp, |
There was a problem hiding this comment.
🔵 minor — Empty filtered comparison is treated as a retention gap, so a genuinely new service loses its new_service flag
requestedCmp comes from baseConfig, which already applies input.services, where and spanKind. So stats.size === 0 also happens when the filter only matches new traffic, for example services:["svc-launched-today"]. In that case the tool falls back to previous_period and then to none, blames retention in comparisonNote, and never reports the new_service flag. Fix: decide the fallback with an unfiltered existence probe on the comparison window (time range only), and keep the requested comparison when the table does have data there.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
| const { summary, services } = parse(await call()); | ||
| expect(services.map(s => [s.service, s.status])).toEqual([ | ||
| [svc('erroring'), 'critical'], | ||
| [svc('slow'), 'degraded'], |
There was a problem hiding this comment.
🔵 minor — Integration test never runs the comparison fallback, new-operation detection, or the novelty lift against real ClickHouse
Every seeded span is named 'GET /api' and both windows have data, so findNewOperations always returns empty and resolveComparison always returns the requested window. A regression that broke the operations SQL, or made the fallback never fire, would still pass. The unit tests stub these inputs (newOperations: new Map()). Fix: add an int case that seeds a span name present only in the current window and checks newOperations plus status:'new_operation'. Add another case with no yesterday data that checks summary.comparison === 'previous_period' and comparisonNote.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
PR Review
4 finding(s): 🔴 0 critical · 🟠 2 major · 🔵 2 minor None could be anchored to changed lines; they are listed below. Findings outside the changed lines
2 minor
Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides. |
Deep Review✅ No critical issues found. This is a carefully written, well-decomposed addition. The five substantive prior-review findings (Greptile) were verified as addressed in the current code: 🟡 P2 -- recommended
🔵 P3 nitpicks (2)
Reviewers (2): previous-comments, orchestrator deep-read (correctness/security/reliability/performance file-by-file). Coverage note: The parallel fan-out (correctness, adversarial, performance, reliability, kieran-typescript, testing, maintainability, project-standards) did not return results within this run's window; findings above come from the completed previous-comments reviewer and the orchestrator's own verification. A re-run is advisable if full multi-persona corroboration is required before merge. Testing gaps: Dedicated unit coverage appears thin for a few pure functions worth pinning down — No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond |
| groups, | ||
| comparisonErrorCounts, | ||
| perService, | ||
| g => !baseline.get(service) || newOps.has(groupValue(g, spanKey) ?? ''), |
There was a problem hiding this comment.
🔵 minor — The rule for a new or growing error group is written twice and the copies must be kept in sync by hand
collect.ts passes g => !baseline.get(service) || newOps.has(groupValue(g, spanKey)) to selectTopErrors. report.ts:204 (changedErrors) rebuilds the same rule as hasBaseline(r) && !onNewOperation(r, g) plus isChanged. If one copy changes and the other does not, the groups listed first will disagree with the new_errors flag and the summary. Fix: export one isNovelError(r, g, comparisonCounts, newOperations, spanKey) from report.ts and use it in both places, or have collectErrors return the changed groups for buildReport to reuse.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
| * Agents often pass a table or source name where the ID belongs. Point them at | ||
| * the exact ID when the input names one trace source, else list the options. | ||
| */ | ||
| async function sourceNotFound( |
There was a problem hiding this comment.
🔵 minor — The hint that maps a name or table to its source ID only helps this one tool
sourceNotFound covers a mistake agents make on any tool that takes a sourceId. Every other tool still returns a bare 'Source not found: … Call clickstack_list_sources' (helpers.ts:476, sql.ts:126, eventDeltas.ts:245, trace/waterfall.ts:349, trace/breakdown.ts:84, runEventPatterns.ts:71). Fix: move it into a shared helper such as mcp/tools/query/helpers.ts with a kind parameter, and call it from those not-found branches.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
| cmp: null, | ||
| comparisonMode: 'none', | ||
| comparisonNote: | ||
| `The "${requestedMode}" window${fallback ? ' and the previous period have' : ' has'} ` + |
There was a problem hiding this comment.
🔵 minor — The empty-window check uses the user's where/services filters, so a filter that only matches new data looks like a retention gap
Scenario: a post-deploy check with where: "ServiceVersion = 'v2'" or services: ["<service launched 40m ago>"]. requestedCmp is empty only because the filter matches nothing yesterday. The tool then falls back to previous_period, and if that is empty too, sets comparisonMode: 'none'. So the service loses new_service, its error groups are never marked new, and the note blames retention. Fix: run the 'does the comparison window have data' probe without input.where/input.services (for example a count() with only the time and spanKind filters), and fall back only when the unfiltered window is empty.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
Deep Review✅ No critical issues found. This is a carefully decomposed, well-tested addition. Eleven reviewers ran; the core ranking/scoring math, novelty detection, comparison-window fallback, cross-window 🟡 P2 -- recommended
🔵 P3 nitpicks (6)
Reviewers (11): correctness, adversarial, security, performance, reliability, kieran-typescript, testing, maintainability, project-standards, api-contract, previous-comments. Testing gaps: No coverage for the comparison-window error fetch truncating at the 2500-row boundary (bounded by No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond |
No trace source we checked has MVs, the error-group queries cannot use one, and the MV rewrite rounds windows to the view's granularity, which broke the half-open comparison windows (review minors).
| // Match groups across windows by a fixed-size hash. Inlining the raw key | ||
| // text (status messages can be stack traces) can exceed max_query_size. | ||
| const hashExpr = `cityHash64(${keyExprs.map(e => `toString(${e})`).join(', ')})`; | ||
| const epochMs = `toUnixTimestamp64Milli(toDateTime64(${getFirstTimestampValueExpression(source.timestampValueExpression)}, 3))`; |
There was a problem hiding this comment.
With a supported timestamp expression such as EventDate, EventTime, fetchErrorGroups uses only EventDate for the first and last error times. Every error that day then appears at midnight. Even errors still arriving at the end of an afternoon report become a stopped burst, and the summary calls them transient.
Use the highest-precision timestamp column, such as the one selected by pickBucketTimestampColumn, for these times.
Knowledge Base Used: Limit trace breakdown queries and explain timeouts
| const onNewOperation = (r: RankedService, g: ErrorGroup) => | ||
| (args.newOperations.get(r.service) ?? []).some( | ||
| o => o.operation === groupValue(g, args.spanNameKey), | ||
| ); |
There was a problem hiding this comment.
Rollouts look like new failures
A supported custom grouping such as errorGroupBy: "StatusMessage" removes the span-name key that onNewOperation needs. Errors entirely from a new operation then receive status: "new" instead of new_operation, add new_errors, and move the service ahead in the results. These false warnings can crowd more useful services out of the report.
Keep the operation information needed for this check separately from the requested display grouping.
## Summary
`clickstack_service_health` no longer tells the agent whether a service is critical, degraded or healthy. A healthy service used to get "report this service as steady, no action needed". That verdict comes from service-wide totals, so a regression confined to one endpoint, pod or attribute segment can sit under every threshold while the tool tells the agent to stop looking. In the `deploy-regression` eval, main calls the service with the bug "Healthy".
Each service now carries only evidence:
| | Before | After |
|---|---|---|
| Per-service field | `status: critical / degraded / healthy` | removed; `flags` and `score` stay |
| Summary, nothing crossed | `Healthy: … normal variance; report this service as steady, no action needed.` | `No metric crossed a threshold vs yesterday: error rate 0.50% vs 0.49%, p95 120ms vs 115ms, requests 3000 vs 2900.` |
| Summary, threshold crossed | `CRITICAL: error rate 8.00% vs 0.50%.` | `Crossed thresholds: error rate 8.00% vs 0.50%.` |
| Report summary | `statusCounts: {critical, degraded, healthy}` | `servicesFlagged` |
| Ordering | status, then new errors, then score | threshold-flagged services, then the rest; within each, new or growing errors first, then score |
| Span attributes | `mcp.service_health.critical_count`, `degraded_count` | `mcp.service_health.flagged_count` |
The tool description and `usage` now say that thresholds apply to service-wide totals and point to `clickstack_table` breakdowns before a service is ruled out. "No flags is a valid result for a routine check" stays, so status reports are not pushed into escalating.
`statusCounts` and per-service `status` are gone from the response. Both are MCP output read by agents; nothing else in the repo consumes them.
## Evals
A/B with `packages/hdx-eval`: branch (slot 99) vs `main` (slot 98), both built from `02f6fd060`, identical seeded data, `claude-opus-4-6`.
| Scenario | Runs/arm | Branch | Main | Δ | Tool calls (branch / main) |
|---|---|---|---|---|---|
| `service-health-check` | 3 | 76% | 75% | +1 | 7.7 / 5.7 |
| `error-root-cause` | 3 | 99% | 98% | +1 | 10.3 / 9.0 |
| `segmented-regression` | 8 | 75.5% | 71.6% | +3.9 | 14.9 / 17.1 |
| `deploy-regression` | 8 | 70.3% | 74.6% | −4.3 | 15.6 / 13.9 |
| `quiet-saturation` | 8 | 84.3% | 88.8% | −4.5 | 19.1 / 16.8 |
| **Average** | | **81.0%** | **81.6%** | **−0.6** | |
The change is neutral on scores. Per-run spread within an arm is 15–20 points, so no difference is significant (the largest has t ≈ 1); `segmented-regression` went from −11 after 3 runs to +13 over the next 5. Routine checks did not get more alarmist: the judge scored `service-health-check` 8 points higher on the branch. Its regex false-alarm checks scored it lower, but they mostly match negated phrases such as "not ongoing" and "not escalation-worthy". Agents make a few more tool calls in most scenarios once the tool stops saying "no action needed".
### How to reproduce
**Starting point:** local full stack (`yarn dev`) with trace data, and an MCP client such as `npx @modelcontextprotocol/inspector` connected to the local MCP endpoint (see `MCP.md`).
**On `main`:**
1. Call `clickstack_list_sources` and copy a `kind: "trace"` source ID.
2. Call `clickstack_service_health` with `{ "sourceId": "<id>" }`.
3. Each service has a `status`, and steady services read "Healthy: … no action needed."
**On this branch:**
1. Repeat the call.
2. There is no `status` field. Summaries state the crossed thresholds or the current vs comparison values, and `summary.servicesFlagged` replaces `statusCounts`.
### How to test on Vercel preview
N/A — non-UI change.
### References
- Linear Issue: HDX-5877
- Related PRs: #3411 (added `clickstack_service_health`)
Summary
Adds
clickstack_service_health, an MCP tool that answers "how are my services doing?" in one call. Today an agent stitches this together from severalclickstack_table/clickstack_timeseriescalls and then has to judge on its own which changes matter.For every service on a trace source, the tool returns throughput, error rate, p50/p95/p99 and the top error groups. Each value is compared with the same window yesterday, and services come back ranked worst first. Each service gets:
critical/degraded/healthy) and the flags behind it (error_rate_up,latency_up,traffic_drop, ...). Flags fire only past fixed thresholds, so routine variance is not reported as a regression.Behavior worth knowing:
spanKind: "all"to include every span).topN, so a quiet service with a brand-new failure still surfaces.The span error predicate moves from
packages/app/src/source.tstopackages/common-utils/src/core/spanPredicates.ts(with a server span-kind predicate) so the API and the app share one definition. No behavior change in the app.Eval results
A/B against
mainwithpackages/hdx-eval, both arms on the same base commit, 5 runs per arm.mainservice-health-checkerror-root-causeservice-health-checkis the use case this tool is for: a routine status report where the agent has to report SLIs, call out what is new, and not escalate routine variance. Agents also used about half the tool calls.error-root-cause: in every branch run, the tool's top result was the service with the new failure (checkout-api, a newpayment_service_unavailableerror group), which points the agent straight at the root cause.How to reproduce
Starting point: local full stack (
yarn dev) with some trace data, and an MCP client (for examplenpx @modelcontextprotocol/inspector) connected to the local MCP endpoint (seeMCP.md).On
main:tools/list. There is noclickstack_service_health.On this branch:
tools/list.clickstack_service_healthis listed.clickstack_list_sourcesand copy theidof akind: "trace"source.clickstack_service_healthwith{ "sourceId": "<id>" }.status,flags,summary, current and comparison metrics, andtopErrors.sourceId(for example"otel_traces"). The error names the correct source ID.To run the evals:
packages/hdx-evalREADME, scenariosservice-health-checkanderror-root-cause.How to test on Vercel preview
N/A — non-UI change.
References