Skip to content

Drew/heatmap promql histogram - #3331

Closed
pulpdrew wants to merge 2 commits into
mainfrom
drew/heatmap-promql-histogram
Closed

pulpdrew wants to merge 2 commits into
mainfrom
drew/heatmap-promql-histogram

Conversation

@pulpdrew

@pulpdrew pulpdrew commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

Summary

This PR adds a new heatmap mode ("histogram") for Prometheus metrics intended to support displaying pre-bucketed histogram-type data as a heatmap.

This is a distribution-type histogram (y-axis is numeric buckets, color is count of observations), but with the buckets pre-computed in the metric. The bounds are taken from the le (less than or equal to) labels on each metric, and the counts are the (de-accumulated) values of the metric.

Users should sum by (le) (...expression) to sum by bounds across series.

Screenshots or video

Screenshot 2026-10-08 at 1 32 44 PM Screenshot 2026-10-08 at 1 57 19 PM

How to test locally

  • Create a promql data source on the metrics_ts timeseries table
  • Create a heatmap chart in histogram mode

References

  • Linear Issue: Closes HDX-5612
  • Related PRs:

@vercel

vercel Bot commented Oct 8, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Oct 8, 2026 5:58pm UTC
hyperdx-storybook Ready Ready Preview Oct 8, 2026 5:58pm UTC

Request Review

@changeset-bot

changeset-bot Bot commented Oct 8, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: cc8eaa1

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages
Name Type
@hyperdx/common-utils Patch
@hyperdx/api Patch
@hyperdx/app Patch
@hyperdx/otel-collector Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@pulpdrew pulpdrew closed this Oct 8, 2026
@github-actions github-actions Bot added the review/tier-4 Critical — deep review + domain expert sign-off label Oct 8, 2026
@github-actions

github-actions Bot commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

🔴 Tier 4 — Critical

Touches authentication, tenancy data models, the public API or shipped database config — or substantially changes the query rendering engine, background tasks, the OTel pipeline, image build, or release CI.

Why this tier:

  • Large diff: 1308 production lines changed (threshold: 1000)
  • Cross-layer change: touches frontend (packages/app) + shared utils (packages/common-utils)

Review process: Deep review from a domain expert. Synchronous walkthrough may be required.
SLA: Schedule synchronous review within 2 business days.

Stats
  • Production files changed: 27
  • Production lines changed: 1308 (+ 1385 in test files, excluded from tier calculation)
  • Branch: drew/heatmap-promql-histogram
  • Author: pulpdrew

To override this classification, remove the review/tier-4 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

@github-actions

github-actions Bot commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

Duplicate code analysis

No new duplicate code introduced by this PR.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 125 in total (1.74%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

return { mode: 'histogram', config };
}
return {
mode: 'calculated',

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 major — PromQL heatmap tiles saved before this change silently switch from series to distribution rendering

Before this PR, convertFormStateToSavedChartConfig stripped heatmap from PromQL tiles; the removed test was 'saves a heatmap ... and no heatmap mode'. Every PromQL heatmap saved that way has no heatmap.mode, so getHeatmapMode now returns 'distribution' and toHeatmapQuery sends it down the new calculated path. Their legend template is also hidden in the drawer. These tiles have only shipped on main so far (.changeset/promql-heatmap.md is still pending), so the cheapest fix is a one-off migration that writes heatmap: { mode: 'series' } onto existing PromQL heatmap tiles. The other option is to make a missing mode resolve to 'series' for PromQL configs only.

Nothing blocks merge automatically, but a maintainer will expect this fixed if it is a real defect in code this PR changes. If it is about surrounding code, reply and say so instead of patching. Do not widen the PR. How to respond

}
return numberFormat
? formatNumber(bound, { ...numberFormat, average: true })
: COMPACT_NUMBER_FORMAT.format(bound);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Bucket bound labels are rounded to 2 significant digits, despite the doc comment

The doc comment says adjacent bounds must not round to the same label, but COMPACT_NUMBER_FORMAT (compact notation) rounds to about 2 significant digits. So 1.05 and 1.1 both render as "1.1", 0.125 renders as "0.13", and 1024 renders as "1K", which collides with a 1000 bound in both the axis and the tooltip range. For unformatted bounds, use a formatter that keeps the bound's significant digits, e.g. Intl.NumberFormat with maximumSignificantDigits: 6, or compact with roundingPriority. Add a test case with a 3-significant-digit bound.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

legendTemplate: z.string().max(MAX_LEGEND_TEMPLATE_LENGTH).optional(),
heatmap: HeatmapSettingsSchema.extend({
mode: PromqlHeatmapModeSchema.optional(),
scaleType: HeatmapScaleTypeSchema.optional(),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — The PromQL heatmap schema allows scaleType with modes that ignore it, and scale type now lives in two places

heatmap.scaleType is only read in distribution mode, yet the schema accepts it alongside series and histogram, and formPromqlHeatmap persists it unchanged. Meanwhile builder heatmaps keep their scale on select[0].heatmapScaleType, so the same concept has two storage locations that EditTimeChartForm must branch on. Option 1: make the PromQL heatmap a discriminated union on mode, with scaleType only on the distribution variant. Option 2: strip scaleType in formPromqlHeatmap when the mode is not distribution.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

);
}

function HistogramHeatmapChart({

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — SeriesHeatmapChart, CalculatedHeatmapChart and HistogramHeatmapChart are copies of each other

The three wrappers differ only in which data hook they call. Factor them into one component that takes the hook result, or pick the hook from query.mode and render a single HeatmapChartView, so the plotKey and numberFormat wiring isn't repeated three times.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

@github-actions

github-actions Bot commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 critical or 🟠 major in code this PR adds or changes: fix before asking for review.
  • 🔵 minor in code this PR changes: your call. Fix if small, otherwise reply.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    severity. Reply with one sentence; if it is critical, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

5 finding(s): 🔴 0 critical · 🟠 1 major · 🔵 4 minor

4 posted as inline comment(s) on the changed lines. 1 listed below.

Findings outside the changed lines

1 minor
  • 🔵 packages/app/src/components/DBHeatmapChart/heatmapGrid.ts:3 — HeatmapScaleTypeSchema is added, but the app type and the API schema still restate the enum by hand → The PR adds HeatmapScaleTypeSchema in common-utils/src/types.ts, but HeatmapScaleType here is still a hand-written 'log' | 'linear', and packages/api/src/utils/zod.ts:522 still has z.enum(['log', 'linear']). Derive both from the new schema: z.infer<typeof HeatmapScaleTypeSchema> here, and HeatmapScaleTypeSchema.optional() in the API, so there is one source of truth.

Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides.

@greptile-apps

greptile-apps Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 4/5

[Medium risk] Adds histogram and calculated heatmap modes to PromQL charts.

Preserve the meaning of existing PromQL heatmaps before merging.

Fix All in Claude CodeFindings

  1. P1 Saved heatmaps change meaning ▶
  2. P2 Higher buckets gain extra counts ▶
  3. P2 Bucket labels lose their limits ▶
  4. P2 Sparse queries do extra work ▶

Summary

Adds PromQL distribution and histogram heatmaps alongside series mode.

  • Histogram mode reads le bounds and subtracts cumulative bucket values.
  • The editor saves heatmap mode and distribution scale settings.
  • Existing PromQL heatmaps need their old series mode preserved.
  • Bucket counts, numeric labels, and large sparse-query work need follow-up.
Diagram
%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Saved PromQL heatmap] --> B{Mode}
  B -->|Series| C[Range query]
  B -->|Distribution| C
  C --> D{Draw mode}
  D -->|Series| E[One row per series]
  D -->|Distribution| F[Count samples in value buckets]
  B -->|Histogram| G[Range query with le labels]
  G --> H[Subtract cumulative bucket values]
  H --> I[One row per bucket]
Loading

Reviews (1) · Last reviewed commit: "feat: Support heatmaps on prometheus his..." · Reviewed by Greptile

Comment on lines +104 to +109
if (isPromqlChartConfig(config)) {
if (mode === 'histogram') {
return { mode: 'histogram', config };
}
return {
mode: 'calculated',

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Saved heatmaps change meaning

Previously saved PromQL heatmaps have no heatmap setting because every PromQL heatmap rendered one row per series. They now fall through to calculated, which merges their series into a distribution. Existing dashboards silently show a different kind of chart after upgrading.

Preserve series mode for old tiles through a read-time migration, and set distribution mode explicitly on new tiles.

Knowledge Base Used:

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Fix in Claude Code Fix in Conductor Fix in Cursor Fix in Codex

Comment on lines +49 to +50
cells[ti * rows + r] = Math.max(0, cum - below);
below = cum;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Higher buckets gain extra counts

Clamping a decreasing bucket to zero still lowers below, so the next bucket gains observations already counted. Cumulative values [5, 4.9, 10] produce [5, 0, 5.1], totaling 10.1 despite the final bucket reporting 10. The extra counts also change the tooltip's percentiles.

Keep below at the greatest cumulative value seen so far, and add a test with a decreasing middle bucket.

Fix in Claude Code Fix in Conductor Fix in Cursor Fix in Codex

Comment on lines +106 to +108
return numberFormat
? formatNumber(bound, { ...numberFormat, average: true })
: COMPACT_NUMBER_FORMAT.format(bound);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Bucket labels lose their limits

When a numeric format omits mantissa, formatBucketBound uses formatNumber's default of zero decimal places. Fractional bounds such as 0.1 and 0.5 become rounded labels instead of their actual limits. Different buckets can share a label, and tooltips can show ranges such as 0 – 0.

Choose enough decimal places to distinguish bucket bounds when the user has not set a precision.

Fix in Claude Code Fix in Conductor Fix in Cursor Fix in Codex

Comment on lines +27 to +29
// Every series counts towards the distribution, so none are dropped. This is a
// client-side limit, so capping it wouldn't affect how much data is in memory.
maxSeries: Number.POSITIVE_INFINITY,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Sparse queries do extra work

The calculated heatmap keeps every series, then passes them through responseToHeatmapSeries, which checks every series at every returned timestamp. Queries with many short-lived series can turn a small response into millions of empty lookups and extra chart objects, making the page unresponsive.

Build the distribution directly from the returned rows. That keeps all samples without first building a full time-chart representation.

Fix in Claude Code Fix in Conductor Fix in Cursor Fix in Codex

@github-actions

github-actions Bot commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

Deep Review

✅ No critical issues found. The new PromQL histogram heatmap mode is correct and well-covered: de-accumulation of cumulative buckets, le-label parsing/sorting (including +Inf, differently-written bounds, non-monotonic clamping, single/empty buckets), and bucket-axis rendering all hold up under both independent tracing and the reviewer fan-out. The findings below are recommendations and nits.

🟡 P2 — recommended

  • packages/app/src/components/DBHeatmapChart/heatmapHistogram.ts:36 — The NaN-le and non-finite-v skip branches of gridFromHistogramSamples are a separately exported, independently unit-tested function but have no direct test for malformed samples.
    • Fix: Add gridFromHistogramSamples cases with an unparseable le (e.g. 'abc'/'') and a non-finite v, asserting the bad sample is dropped while other buckets de-accumulate correctly.
    • testing, correctness, adversarial
  • packages/app/src/components/DBEditTimeChartForm/EditTimeChartForm.tsx:417 — The PromQL heatmap glue — storedHeatmapMode vs builderHeatmapMode fallback, heatmapScaleType field routing (heatmap.scaleType for PromQL vs series.0.heatmapScaleType for builder), and applyHeatmapDefaults's isPromql duration-format suppression — has no integration test, so a regression writing the wrong field would go uncaught.
    • Fix: Add a rendered-form test switching a PromQL heatmap across distribution/series/histogram and asserting scale-type persists to the correct field and no duration format is applied.
    • testing, correctness, maintainability
🔵 P3 nitpicks (7)
  • packages/common-utils/src/core/promql.ts:142 — parsePromqlLe special-cases only the exact string '+Inf': a non-standard 'Inf'/'inf' label parses to NaN and throws, blanking the whole chart, while 'Infinity' is accepted as a second numeric-infinity bound that would be summed with '+Inf' and double-count the top bucket.
    • Fix: Normalize infinity labels case-insensitively and treat only the canonical open bucket as Infinity.
    • adversarial
  • packages/app/src/components/DBHeatmapChart/heatmapHistogram.ts:49 — When an intermediate bucket is absent at a time column, below carries the last present cumulative, so the skipped bucket's mass is attributed to the next present row (labeled with only its own narrower range); tested as intended but can mislabel counts if a scrape drops a bucket.
    • Fix: Consider carrying NaN gaps forward as unknown rather than folding skipped mass into the next present bucket, or document the attribution in the axis/tooltip.
    • correctness, adversarial
  • packages/app/src/components/DBHeatmapChart/useHistogramHeatmapData.ts:24 — Histogram rows are read via untyped magic-string keys (row.le, row.__hdx_time_bucket, row.value) with no shared contract, so a query column rename would break silently at runtime.
    • Fix: Introduce a typed row interface or shared column-name constants for the histogram query's response shape.
    • maintainability, kieran-typescript
  • packages/app/src/components/DBHeatmapChart/useHistogramHeatmapData.ts:25 — The __hdx_time_bucket column name is hardcoded here (and in toHistogramRows) instead of the exported FIXED_TIME_BUCKET_EXPR_ALIAS, giving the alias two sources of truth.
    • Fix: Import and use FIXED_TIME_BUCKET_EXPR_ALIAS at both the producer and consumer.
    • maintainability
  • packages/app/src/components/DBEditTimeChartForm/HeatmapModeControl.tsx:49 — isOfferedMode(value): value is M narrows based on allowHistogram-selected options, which is decoupled from the generic M; both current call sites are consistent, but a future caller with mismatched mode/allowHistogram would get an unsound onModeChange.
    • Fix: Derive the offered modes from a parameter typed to M, or constrain the component so options and M cannot diverge.
    • kieran-typescript
  • packages/app/src/components/DBHeatmapChart/useContentStable.ts:10 — isEqual compares functions/class instances by reference, so a value carrying a non-plain member would flip isSame false every render and self-trigger an unbounded setStable loop; not reachable with today's plain-data callers.
    • Fix: Document the plain-JSON-only contract (or add a dev-only render-count guard) before the hook is reused for richer objects.
    • julik-frontend-races
  • .changeset/nice-cows-hug.md:3 — Declares a @hyperdx/api patch bump, but no packages/api/ file is changed anywhere in this PR.
    • Fix: Drop the @hyperdx/api entry unless an api behavior change is actually intended.
    • project-standards

Reviewers (10): correctness, adversarial, testing, maintainability, performance, kieran-typescript, julik-frontend-races, project-standards, agent-native, learnings-researcher.

Testing gaps:

  • heatmapYAxisOptions bucket-axis branch (scale range: [0, bounds.length] and the v - 1 bound-label mapping) has no direct test; only its sub-helpers are tested in isolation.
  • DBHeatmapChart mode dispatch (new calculated/histogram branches) and HeatmapPlot bucket rendering (boundFormatter, kind: 'bucket' tooltip, computeBucketPercentiles) are verified only via unit helpers, not integration.
  • No test covers a single-time-column histogram (times.length === 1, stepMs === 0) where sample timestamps don't exactly match the generated bucket, which would silently render an empty grid.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

@github-actions

github-actions Bot commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 409 passed • 1 skipped • 1723s

Status Count
✅ Passed 409
❌ Failed 0
⚠️ Flaky 1
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

This branch was successfully deployed

2 active deployments
Preview – hyperdx-oss — cc8eaa15 Deployed Oct 8, 2026 by vercel[bot]
Preview – hyperdx-storybook — cc8eaa15 Deployed Oct 8, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review/tier-4 Critical — deep review + domain expert sign-off

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant