Repository navigation
fix(app): count legacy STATUS_CODE_ERROR spans as errors - #3181
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
🦋 Changeset detectedLatest commit: 2f5ac48 The changes in this PR will be included in the next version bump. This PR includes changesets to release 3 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🔵 Tier 2 — Low RiskSmall, isolated change with no API route or data model modifications. Why this tier:
Review process: AI review + quick human skim (target: 5–15 min). Reviewer validates AI assessment and checks for domain-specific concerns. Stats
|
|
Deep Review✅ No critical issues found. This diff centralizes the errored-span SQL predicate into one exported helper, 🔵 P3 nitpicks (1)
Reviewers (2): correctness, testing. Testing gaps: The exported No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond |
| * ('STATUS_CODE_ERROR'); severity-text values never take the latter, so it is | ||
| * a no-op there. | ||
| */ | ||
| export function errorPredicateSql(statusOrSeverityExpression: string): string { |
There was a problem hiding this comment.
🔵 minor — errorPredicateSql mixes the span-status and log-severity vocabularies, so it is wrong in both directions on the log path
The single value set ('error', 'status_code_error') is applied both to span status codes and, at llm/lib/expressions.ts:487, to source.severityTextExpression; status_code_error is unreachable for severity text (the doc comment says so), while the values severity text actually takes for errors — OTel's FATAL, plus the crit/alert/emerg/err* prefixes that the app's own getLogLevelClass (packages/app/src/utils.ts:332) already treats as errors — are still excluded, so a FATAL log renders red in the table but is not counted by the LLM Errors tab. Either split this into two helpers (span status vs. log severity) or derive the severity arm from the same prefix list getLogLevelClass uses, so the SQL definition and the JS classifier cannot drift.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
There was a problem hiding this comment.
Fixed in 7de8c84: the helper is now errorStatusPredicateSql, documented as span-status only, and the log call site keeps its own predicate unchanged instead of borrowing the status vocabulary. Widening log severity to fatal/crit/alert/emerg is a pre-existing gap rather than something this PR introduced, so it is noted in #3203 with the other remaining spellings.
PR Review
1 finding(s): 🔴 0 critical · 🟠 0 major · 🔵 1 minor None could be anchored to changed lines; they are listed below. Findings outside the changed lines1 minor
Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides. |
E2E Test Results✅ All tests passed • 408 passed • 1 skipped • 1491s
Tests ran across 4 shards in parallel. |
The service dashboard and the LLM expressions each spelled out their own "is this an error" predicate as lower(<status>) = 'error'. Collectors that emit the long OTel form (STATUS_CODE_ERROR) therefore fell outside it, so error rates and error counts on those dashboards undercounted. Pull the predicate into one exported helper in source.ts and match both forms. The logs path passes a severity-text expression, which never takes the long form, so it is unaffected.
90a6fdc to
a690c51
Compare
The helper was also applied to the log-severity expression, where the legacy status value it matches can never appear and the values severity text does use for errors (fatal, crit and friends) are a different set. Mixing the two vocabularies in one predicate made it wrong in both directions on the log path. Rename it to errorStatusPredicateSql, document it as span-status only, and let the log caller keep its own predicate, unchanged from before this branch.
| * column, and the values severity text does use for errors are a | ||
| * different set. | ||
| */ | ||
| isError: `lower(${severityText}) = 'error'`, |
There was a problem hiding this comment.
🔵 minor — New comment says severity text uses a different set of error values, but the predicate still matches only 'error'
The new comment (and the one on errorStatusPredicateSql in source.ts) says severity-text errors are a different set ('fatal', 'crit' and so on). The code right below it is still lower(severityText) = 'error', so fatal and critical log events are left out of the LLM log error count. Either widen the predicate to the values the comment lists, or reword the comment so it only explains why the span-status helper isn't reused here.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
|
On the service map finding: agreed it is a real gap and it is the sharpest of the set, since after this PR the map and the Services dashboard disagree on the same spans. Both sites are outside this PR's files though, so rather than widen this change I filed #3203, which covers the service map, its tooltip drill-down, and the client-side check in the trace waterfall (that one needs a JS-side predicate, not the SQL helper). Flagging it here so a maintainer can pull it forward if they would rather it ship together. |
Duplicate code analysisNo new duplicate code introduced by this PR. jscpd compared this branch against |
## Summary Service map error counts, the error drill-down, and the trace waterfall now treat legacy `STATUS_CODE_ERROR` span statuses the same way the services dashboard already does. Collectors that still write that value no longer show a service or trace as error-free. ## Root cause `errorStatusPredicateSql` already matches both `Error` and `STATUS_CODE_ERROR`. The service map still counted only `ServerSpans.statusCode = 'Error'`, and its error drill-down used the same exact match. The waterfall's `getRowSeverity` returned an error only when `StatusCode === 'Error'`. On a deployment whose collector writes `STATUS_CODE_ERROR`, those surfaces reported zero errors while the services dashboard did not. Fixes #3203. ## Fix The SQL predicate and a new `isErrorStatusCode` helper share one list: `error` and `status_code_error`, compared case-insensitively. The service map count and the error drill-down use the SQL predicate. The waterfall uses the helper for the error count and the row highlight. Log severity is unchanged. ## Test plan - Before the helper accepted legacy values, `jest --testPathPatterns spanPredicates` failed 5 cases (`ERROR`, `error`, `STATUS_CODE_ERROR`, `status_code_error`, `Status_Code_Error` expected true, received false) when `isErrorStatusCode` was an exact `=== 'Error'` check. - After the fix, from `packages/common-utils`: `node ..\..\.yarn\releases\yarn-4.13.0.cjs exec jest --coverage=false --testPathPatterns spanPredicates` passed, 17 tests. - From the repo root: `node .yarn/releases/yarn-4.13.0.cjs workspace @hyperdx/app exec jest --coverage=false --testPathPatterns "useServiceMap.test|ServiceMap/__tests__/utils.test"` passed, 86 tests. - `node .yarn/releases/yarn-4.13.0.cjs workspace @hyperdx/common-utils build` succeeded. - `eslint --flag v10_config_lookup_from_file --quiet` on the touched files exited 0. Prettier was run on those files. ### How to reproduce **Starting point:** a trace source whose status column contains `STATUS_CODE_ERROR` (legacy collector) as well as `Error`. The public demo dataset does not store that legacy value, so the unit tests above are the reliable reproduction. **On `main`:** 1. Open the service map for that source. Nodes whose spans use only `STATUS_CODE_ERROR` show 0.00% errors. 2. Open a trace that contains a `STATUS_CODE_ERROR` span. The waterfall does not mark that span as an error, and the header error count skips it. **On this branch:** 1. The same service map counts those spans, and the error drill-down filter is `lower(StatusCode) IN ('error', 'status_code_error')`. 2. The waterfall highlights those spans and includes them in the error count. ### Screenshots or video No screenshots. The change is the status match used for counts and highlighting, not layout. Reproducing the visible difference needs rows stored as `STATUS_CODE_ERROR`, which play.hyperdx.io does not provide. ### How to test on Vercel preview N/A. This is not a layout change. The corrected counts show up only when the selected trace source stores `STATUS_CODE_ERROR`. ### References - Fixes #3203 - Related PR: #3181 (services and LLM dashboards)
Summary
The service dashboard and the LLM expressions each spelled out their own "is this an error" predicate as
lower(<status>) = 'error'. Both are fedsource.statusCodeExpressionfor trace sources (serviceDashboard.ts:164,llm/lib/expressions.ts:443), so a collector that writes the long OTel form,STATUS_CODE_ERROR, fell outside the predicate and those spans went uncounted. Error rate and error counts on the Services and LLM dashboards undercounted as a result.This pulls the definition into one exported helper,
errorStatusPredicateSqlinsource.ts, and matches both the shortErrorform and the longSTATUS_CODE_ERRORone.The helper is span-status only. Log severity text is a different vocabulary: the long status value can never appear in it, and the values severity text does use for errors (
fatal,critand friends) are a different set again. So the log call site (expressions.ts:487) keeps a predicate of its own, unchanged from before this branch, rather than borrowing the span-status one.A single definition for span status also means the next place that needs it does not add another spelling.
Test plan
expressions.test.tsandserviceDashboard.test.tsto the widened predicate; both suites pass. Both of those assertions cover trace sources, which is the path that changes.tsc --noEmitandeslintwith no new warnings.No UI or layout change: this only widens which spans an existing query counts as errors. [ui-check: allow]
Notes
patchchangeset.useServiceMap.tsx:178,ServiceMapTooltip.tsx:81) and the client-side check inDBTraceWaterfallChart.tsx. Both are outside this PR's files, and the waterfall one needs a JS-side predicate rather than SQL, so they are tracked in Route the remaining error-status checks through the shared predicate #3203 instead of widening this change.