Skip to content

feat: Add distribution heatmaps calculated from PromQL values - #3326

Merged
pulpdrew merged 2 commits into
mainfrom
drew/heatmap-promql-distribution-calc
Oct 13, 2026
Merged

pulpdrew merged 2 commits into
mainfrom
drew/heatmap-promql-distribution-calc

Conversation

@pulpdrew

@pulpdrew pulpdrew commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Summary

This PR adds support for heatmaps that display a distribution of data queried from PromQL. The distribution is calculated client-side by bucketing the values returned at each timestamp into linear- or log-spaced buckets. The number of samples per bucket determines the color.

Screenshots or video

Show distribution (# of cores) at varying utilization % over time:

sum by (system_cpu_logical_number) (system_cpu_utilization{system_cpu_state!="idle"})
Screenshot 2026-10-08 at 9 53 51 AM

How to test locally

  • Create a promql source on the metrics_ts table
  • Create a distribution mode heatmap

References

  • Linear Issue: Closes HDX-5609
  • Related PRs:

@changeset-bot

changeset-bot Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 98b69d5

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages
Name Type
@hyperdx/common-utils Patch
@hyperdx/api Patch
@hyperdx/app Patch
@hyperdx/otel-collector Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercel Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Oct 13, 2026 11:26pm UTC
hyperdx-storybook Ready Ready Preview Oct 13, 2026 11:26pm UTC

Request Review

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

🟡 Tier 3 — Standard

Introduces new logic, modifies core functionality, or touches areas with non-trivial risk.

Why this tier:

  • Diff size: 648 production lines changed (Tier 2 max: < 250)
  • Cross-layer change: touches frontend (packages/app) + shared utils (packages/common-utils)

Review process: Full human review — logic, architecture, edge cases.
SLA: First-pass feedback within 1 business day.

Stats
  • Production files changed: 16
  • Production lines changed: 648 (+ 691 in test files, excluded from tier calculation)
  • Branch: drew/heatmap-promql-distribution-calc
  • Author: pulpdrew

To override this classification, remove the review/tier-3 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Duplicate code analysis

No new duplicate code introduced by this PR.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 125 in total (1.74%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

Comment thread packages/common-utils/src/types.ts Resolved
Comment thread packages/common-utils/src/types.ts Resolved
@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 critical or 🟠 major in code this PR adds or changes: fix before asking for review.
  • 🔵 minor in code this PR changes: your call. Fix if small, otherwise reply.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    severity. Reply with one sentence; if it is critical, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

4 finding(s): 🔴 0 critical · 🟠 1 major · 🔵 3 minor

4 posted as inline comment(s) on the changed lines.


Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides.

@greptile-apps

greptile-apps Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; no new actionable issue was found.

Summary

Adds PromQL distribution heatmaps that count samples in client-side linear or log buckets.

  • PromQL heatmaps can show how sample values spread over time.
  • PromQL heatmap editors let people choose a mode and scale.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[PromQL heatmap] --> B{Mode}
  B -->|Series| C[Draw one row per series]
  B -->|Distribution| D[Count samples in value buckets]
  D --> E[Linear or log scale]
  C --> F[Heatmap]
  E --> F
Loading

Reviews (6) · Last reviewed commit: "review: Faster rounding, dynamic heatmap..." · Reviewed by Greptile

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

✅ No critical issues found.

This PR adds a client-side "calculated" distribution heatmap mode for PromQL range queries (bucketing every series' sample values into linear/log value buckets per time column). The new bucketing math in heatmapCalculate.ts is well-covered by unit tests (constant values, sub-1e-6 ranges, ranges wider than 1e17, near-1e300 magnitudes, log clamping, empty/non-positive inputs), the useMemo dependency chains correctly key on primitive ms values to avoid needless grid rebuilds, and the previously-raised P1 concerns were either resolved in-branch (dynamic targetBuckets via sqrt clamp, epsilon-floor yStepIndex replacing per-sample lodash.round, the log-scale bucket-count test reconciled to CALCULATED_MAX_BUCKETS) or legitimately out of scope (PromQL heatmaps are unreleased private preview, so routing mode-less saves to calculated has no persisted population to migrate). The findings below are a scalability risk and a state-preservation nit.

🟡 P2 -- recommended

  • packages/app/src/components/DBHeatmapChart/useCalculatedHeatmapData.ts:29 -- Calculated mode sets maxSeries: Number.POSITIVE_INFINITY and runs gridFromSamples synchronously inside a render-phase useMemo, so a large PromQL response (unbounded here; only the API's 100k-series limit applies) does two full passes plus an O(n log n) Float64Array.sort() of every value on the main thread, which can stall the tab for seconds.
    • Fix: bound or downsample the sample count before bucketing (with the planned user-facing notice), or move gridFromSamples off the main thread via a worker or chunked scheduler.
    • performance
🔵 P3 nitpicks (2)
  • packages/app/src/components/DBEditTimeChartForm/EditTimeChartForm.tsx:205 -- For a PromQL heatmap (isPromql true), switching mode skips the builder default but still enters the else if branch that clears numberFormat whenever it equals HEATMAP_DURATION_NUMBER_FORMAT, discarding a duration format a user legitimately chose for a millisecond-valued PromQL metric.

    • Fix: scope the duration-format cleanup to the builder-default application path so the PromQL branch preserves an explicitly chosen format.
  • packages/app/src/components/DBHeatmapChart/heatmapCalculate.ts:147 -- The log-scale path allocates a Float64Array sized to the total point count on every recompute before trimming with subarray(0, count); it is transient and GC'd, but the allocation scales with the uncapped sample count.

    • Fix: no change required now; revisit alongside the sample-count cap.
    • performance

Reviewers (2): performance, orchestrator direct analysis (correctness, previous-comments, api-contract/schema, and editor-wiring lenses applied against the full changed files and the 25 prior PR comments).

Testing gaps: No test exercises gridFromSamples at the stated large scale (hundreds of thousands to millions of samples), so a regression in per-sample or sort cost on the main thread would go unmeasured.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 408 passed • 1 skipped • 1135s

Status Count
✅ Passed 408
❌ Failed 0
⚠️ Flaky 1
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

} from './heatmapGrid';
import { HeatmapSeries, makeTimeIndexer } from './heatmapSeriesGrid';

export const CALCULATED_TARGET_BUCKETS = 10;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could the row count grow with the number of samples instead of staying fixed at 10?

Builder heatmaps use 80 rows. With 10, high-cardinality data loses most of its shape: 10k series of with latency at different extremes would lose any detail indicating tail-end issues. Few series still need few rows, so something like this keeps the screenshot case as it is:

const n = clamp(Math.round(Math.sqrt(maxSamplesPerColumn)), CALCULATED_TARGET_BUCKETS, HEATMAP_N_BUCKETS);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea, updated! And I've added a followup ticket to support custom row counts so the user can choose


/** Index of the `yStep`-tall row holding `value`. */
function yStepIndex(value: number, yStep: number) {
return Math.floor(round(value / yStep, 9));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would a plain epsilon work here instead of lodash round?

round runs once per sample and converts the number to a string and back. For 2.45M samples that's ~930ms against ~12ms for the version below, which is most of the cost of the linear grid:

return Math.floor(value / yStep + 1e-9);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice suggestion - updated!

enabled,
// Every series counts towards the distribution, so none are dropped. This is a
// client-side limit, so capping it wouldn't affect how much data is in memory.
maxSeries: Number.POSITIVE_INFINITY,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should there be an upper limit on samples, now that there's no series cap? Would a sample limit with a notice, like MAX_TABLE_ROWS for tables, make sense here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is an upper limit on the API side (100k series). If it's alright with you, I can add the warning as a followup because it turns out to be a wider change in a variety of places.

@jordan-simonovski

Copy link
Copy Markdown
Contributor

Left you a few questions. Happy to discuss if they don't make sense!

Comment thread packages/common-utils/src/types.ts Resolved
@pulpdrew
pulpdrew force-pushed the drew/heatmap-promql-distribution-calc branch from f1fd37c to 6ca4357 Compare October 13, 2026 12:19
Comment thread packages/common-utils/src/types.ts Resolved
Comment thread packages/common-utils/src/types.ts Resolved
Comment thread packages/common-utils/src/types.ts Resolved
@jordan-simonovski

Copy link
Copy Markdown
Contributor

LGTM! Nice work!

@pulpdrew
pulpdrew force-pushed the drew/heatmap-promql-distribution-calc branch from 1a191fd to 98b69d5 Compare October 13, 2026 23:21
@pulpdrew
pulpdrew merged commit 5cabc11 into main Oct 13, 2026
29 checks passed
@kodiakhq
kodiakhq Bot deleted the drew/heatmap-promql-distribution-calc branch October 13, 2026 23:48

This branch was successfully deployed

2 active deployments
Preview – hyperdx-oss — 98b69d5c Deployed Oct 13, 2026 by vercel[bot]
Preview – hyperdx-storybook — 98b69d5c Deployed Oct 13, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

automerge review/tier-3 Standard — full human review required

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants