Skip to content

fix(app): count legacy STATUS_CODE_ERROR spans as errors - #3181

Merged
kodiakhq[bot] merged 3 commits into
mainfrom
alex/error-span-predicate
Oct 14, 2026
Merged

kodiakhq[bot] merged 3 commits into
mainfrom
alex/error-span-predicate

Conversation

@alex-fedotyev

@alex-fedotyev alex-fedotyev commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The service dashboard and the LLM expressions each spelled out their own "is this an error" predicate as lower(<status>) = 'error'. Both are fed source.statusCodeExpression for trace sources (serviceDashboard.ts:164, llm/lib/expressions.ts:443), so a collector that writes the long OTel form, STATUS_CODE_ERROR, fell outside the predicate and those spans went uncounted. Error rate and error counts on the Services and LLM dashboards undercounted as a result.

This pulls the definition into one exported helper, errorStatusPredicateSql in source.ts, and matches both the short Error form and the long STATUS_CODE_ERROR one.

The helper is span-status only. Log severity text is a different vocabulary: the long status value can never appear in it, and the values severity text does use for errors (fatal, crit and friends) are a different set again. So the log call site (expressions.ts:487) keeps a predicate of its own, unchanged from before this branch, rather than borrowing the span-status one.

A single definition for span status also means the next place that needs it does not add another spelling.

Test plan

  • Updated the existing expectations in expressions.test.ts and serviceDashboard.test.ts to the widened predicate; both suites pass. Both of those assertions cover trace sources, which is the path that changes.
  • Full app suite green (246 files, 4035 tests), plus tsc --noEmit and eslint with no new warnings.

No UI or layout change: this only widens which spans an existing query counts as errors. [ui-check: allow]

Notes

  • Adds a patch changeset.
  • Two other places still carry their own error-status spelling: the service map (useServiceMap.tsx:178, ServiceMapTooltip.tsx:81) and the client-side check in DBTraceWaterfallChart.tsx. Both are outside this PR's files, and the waterfall one needs a JS-side predicate rather than SQL, so they are tracked in Route the remaining error-status checks through the shared predicate #3203 instead of widening this change.

@vercel

vercel Bot commented Sep 22, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Oct 14, 2026 11:56am UTC
hyperdx-storybook Ready Ready Preview Oct 14, 2026 11:56am UTC

Request Review

@changeset-bot

changeset-bot Bot commented Sep 22, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 2f5ac48

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages
Name Type
@hyperdx/app Patch
@hyperdx/api Patch
@hyperdx/otel-collector Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions github-actions Bot added the review/tier-2 Low risk — AI review + quick human skim label Sep 22, 2026
@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

🔵 Tier 2 — Low Risk

Small, isolated change with no API route or data model modifications.

Why this tier:

  • Standard feature/fix — introduces new logic or modifies core functionality

Review process: AI review + quick human skim (target: 5–15 min). Reviewer validates AI assessment and checks for domain-specific concerns.
SLA: Resolve within 4 business hours.

Stats
  • Production files changed: 3
  • Production lines changed: 29 (+ 10 in test files, excluded from tier calculation)
  • Branch: alex/error-span-predicate
  • Author: alex-fedotyev

To override this classification, remove the review/tier-2 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

@greptile-apps

greptile-apps Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

[Medium impact] The PR appears safe to merge with no actionable correctness, security, or repository-rule issues identified.

Summary

Centralizes the SQL predicate used to classify errored spans and expands it to recognize both Error and the legacy STATUS_CODE_ERROR value.

  • Applies the shared predicate to Services dashboard queries and LLM trace expressions.
  • Keeps log severity classification separate because it uses a different vocabulary.
  • Updates focused tests and adds an app patch changeset.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Trace source status expression] --> B[errorStatusPredicateSql]
  B --> C["lower(status) IN ('error', 'status_code_error')"]
  C --> D[Services dashboard error metrics]
  C --> E[LLM trace error metrics]
  F[Log severity expression] --> G["Existing log-specific predicate"]
Loading

Reviews (4) · Last reviewed commit: "Merge branch 'main' into alex/error-span..." · Reviewed by Greptile

@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

✅ No critical issues found.

This diff centralizes the errored-span SQL predicate into one exported helper, errorStatusPredicateSql in packages/app/src/source.ts, and widens it from lower(x) = 'error' to lower(x) IN ('error', 'status_code_error'). The widened predicate is a strict superset of the old one, so no span previously counted as an error stops counting — the change only adds matches for the long OTel STATUS_CODE_ERROR form. Both trace call sites (serviceDashboard.ts:229, llm/lib/expressions.ts:457) pass a status-code expression, the log predicate is left narrow by design, no import cycle is introduced, and both call-site tests were updated to the widened literal.

🔵 P3 nitpicks (1)
  • packages/app/src/llm/__tests__/expressions.test.ts:359 — The intentional divergence where getLLMLogExpressions.isError stays lower(SeverityText) = 'error' (rather than reusing the widened span helper) is documented in code but not asserted by any test, so a future change that widens or re-points the log predicate would pass silently.
    • Fix: Add expect(expressions.isError).toBe("lower(SeverityText) = 'error'") to the existing getLLMLogExpressions describe block to lock in the span-vs-log predicate divergence.

Reviewers (2): correctness, testing.

Testing gaps: The exported errorStatusPredicateSql helper has no dedicated unit test, but both trace call sites assert its exact output string, so a regression reverting the widening is already covered — not a substantive gap.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

Comment thread packages/app/src/source.ts Outdated
* ('STATUS_CODE_ERROR'); severity-text values never take the latter, so it is
* a no-op there.
*/
export function errorPredicateSql(statusOrSeverityExpression: string): string {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — errorPredicateSql mixes the span-status and log-severity vocabularies, so it is wrong in both directions on the log path

The single value set ('error', 'status_code_error') is applied both to span status codes and, at llm/lib/expressions.ts:487, to source.severityTextExpression; status_code_error is unreachable for severity text (the doc comment says so), while the values severity text actually takes for errors — OTel's FATAL, plus the crit/alert/emerg/err* prefixes that the app's own getLogLevelClass (packages/app/src/utils.ts:332) already treats as errors — are still excluded, so a FATAL log renders red in the table but is not counted by the LLM Errors tab. Either split this into two helpers (span status vs. log severity) or derive the severity arm from the same prefix list getLogLevelClass uses, so the SQL definition and the JS classifier cannot drift.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 7de8c84: the helper is now errorStatusPredicateSql, documented as span-status only, and the log call site keeps its own predicate unchanged instead of borrowing the status vocabulary. Widening log severity to fatal/crit/alert/emerg is a pre-existing gap rather than something this PR introduced, so it is noted in #3203 with the other remaining spellings.

@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 critical or 🟠 major in code this PR adds or changes: fix before asking for review.
  • 🔵 minor in code this PR changes: your call. Fix if small, otherwise reply.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    severity. Reply with one sentence; if it is critical, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

1 finding(s): 🔴 0 critical · 🟠 0 major · 🔵 1 minor

None could be anchored to changed lines; they are listed below.

Findings outside the changed lines

1 minor
  • 🔵 packages/app/src/source.ts:126 — Shared span-error predicate lives in an app module that pulls in React and the API client, so the API copies can't use it → Move errorStatusPredicateSql into @hyperdx/common-utils (e.g. core/utils). It is a pure string builder, but source.ts imports React, react-query, @mantine/notifications, @/api and @/localStore. Two problems follow. First, llm/lib/expressions.ts used to depend only on common-utils types and sibling files, and now loads that whole runtime graph. Second, packages/api can't import it. That matters because packages/api/src/mcp/tools/trace/waterfall.ts:407 has the reverse spelling of this bug (${statusCodeExpr} = 'STATUS_CODE_ERROR', which misses the short Error form). packages/app/src/components/aiSummarize/logic.ts:83 also matches only STATUS_CODE_ERROR. Neither is in the follow-ups the PR lists. With the helper in common-utils, the MCP first_error filter can use it, and the doc comment's "single definition" claim can actually hold.

Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides.

@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 408 passed • 1 skipped • 1491s

Status Count
✅ Passed 408
❌ Failed 0
⚠️ Flaky 3
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

The service dashboard and the LLM expressions each spelled out their own
"is this an error" predicate as lower(<status>) = 'error'. Collectors that
emit the long OTel form (STATUS_CODE_ERROR) therefore fell outside it, so
error rates and error counts on those dashboards undercounted.

Pull the predicate into one exported helper in source.ts and match both
forms. The logs path passes a severity-text expression, which never takes
the long form, so it is unaffected.
The helper was also applied to the log-severity expression, where the legacy
status value it matches can never appear and the values severity text does use
for errors (fatal, crit and friends) are a different set. Mixing the two
vocabularies in one predicate made it wrong in both directions on the log path.

Rename it to errorStatusPredicateSql, document it as span-status only, and let
the log caller keep its own predicate, unchanged from before this branch.
* column, and the values severity text does use for errors are a
* different set.
*/
isError: `lower(${severityText}) = 'error'`,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — New comment says severity text uses a different set of error values, but the predicate still matches only 'error'

The new comment (and the one on errorStatusPredicateSql in source.ts) says severity-text errors are a different set ('fatal', 'crit' and so on). The code right below it is still lower(severityText) = 'error', so fatal and critical log events are left out of the LLM log error count. Either widen the predicate to the values the comment lists, or reword the comment so it only explains why the span-status helper isn't reused here.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

@alex-fedotyev

Copy link
Copy Markdown
Contributor Author

On the service map finding: agreed it is a real gap and it is the sharpest of the set, since after this PR the map and the Services dashboard disagree on the same spans. Both sites are outside this PR's files though, so rather than widen this change I filed #3203, which covers the service map, its tooltip drill-down, and the client-side check in the trace waterfall (that one needs a JS-side predicate, not the SQL helper). Flagging it here so a maintainer can pull it forward if they would rather it ship together.

@github-actions

Copy link
Copy Markdown
Contributor

Duplicate code analysis

No new duplicate code introduced by this PR.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 125 in total (1.74%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

@kodiakhq
kodiakhq Bot merged commit f33fc83 into main Oct 14, 2026
29 checks passed
@kodiakhq
kodiakhq Bot deleted the alex/error-span-predicate branch October 14, 2026 12:01
kodiakhq Bot pushed a commit that referenced this pull request Nov 3, 2026
## Summary

Service map error counts, the error drill-down, and the trace waterfall now treat legacy `STATUS_CODE_ERROR` span statuses the same way the services dashboard already does. Collectors that still write that value no longer show a service or trace as error-free.

## Root cause

`errorStatusPredicateSql` already matches both `Error` and `STATUS_CODE_ERROR`. The service map still counted only `ServerSpans.statusCode = 'Error'`, and its error drill-down used the same exact match. The waterfall's `getRowSeverity` returned an error only when `StatusCode === 'Error'`. On a deployment whose collector writes `STATUS_CODE_ERROR`, those surfaces reported zero errors while the services dashboard did not.

Fixes #3203.

## Fix

The SQL predicate and a new `isErrorStatusCode` helper share one list: `error` and `status_code_error`, compared case-insensitively. The service map count and the error drill-down use the SQL predicate. The waterfall uses the helper for the error count and the row highlight. Log severity is unchanged.

## Test plan

- Before the helper accepted legacy values, `jest --testPathPatterns spanPredicates` failed 5 cases (`ERROR`, `error`, `STATUS_CODE_ERROR`, `status_code_error`, `Status_Code_Error` expected true, received false) when `isErrorStatusCode` was an exact `=== 'Error'` check.
- After the fix, from `packages/common-utils`: `node ..\..\.yarn\releases\yarn-4.13.0.cjs exec jest --coverage=false --testPathPatterns spanPredicates` passed, 17 tests.
- From the repo root: `node .yarn/releases/yarn-4.13.0.cjs workspace @hyperdx/app exec jest --coverage=false --testPathPatterns "useServiceMap.test|ServiceMap/__tests__/utils.test"` passed, 86 tests.
- `node .yarn/releases/yarn-4.13.0.cjs workspace @hyperdx/common-utils build` succeeded.
- `eslint --flag v10_config_lookup_from_file --quiet` on the touched files exited 0. Prettier was run on those files.

### How to reproduce

**Starting point:** a trace source whose status column contains `STATUS_CODE_ERROR` (legacy collector) as well as `Error`. The public demo dataset does not store that legacy value, so the unit tests above are the reliable reproduction.

**On `main`:**

1. Open the service map for that source. Nodes whose spans use only `STATUS_CODE_ERROR` show 0.00% errors.
2. Open a trace that contains a `STATUS_CODE_ERROR` span. The waterfall does not mark that span as an error, and the header error count skips it.

**On this branch:**

1. The same service map counts those spans, and the error drill-down filter is `lower(StatusCode) IN ('error', 'status_code_error')`.
2. The waterfall highlights those spans and includes them in the error count.

### Screenshots or video

No screenshots. The change is the status match used for counts and highlighting, not layout. Reproducing the visible difference needs rows stored as `STATUS_CODE_ERROR`, which play.hyperdx.io does not provide.

### How to test on Vercel preview

N/A. This is not a layout change. The corrected counts show up only when the selected trace source stores `STATUS_CODE_ERROR`.

### References

- Fixes #3203
- Related PR: #3181 (services and LLM dashboards)

This branch was successfully deployed

2 active deployments
Preview – hyperdx-oss — 2f5ac484 Deployed Oct 14, 2026 by vercel[bot]
Preview – hyperdx-storybook — 2f5ac484 Deployed Oct 14, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

automerge review/tier-2 Low risk — AI review + quick human skim

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants