Repository navigation
fix: render percentile aggFns as quantile with a level #3357
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
b1f8992
7964968
2ea6b19
aaa63f0
fa85909
86410be
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| --- | ||
| '@hyperdx/common-utils': patch | ||
| '@hyperdx/api': patch | ||
| '@hyperdx/app': patch | ||
| --- | ||
|
|
||
| fix: render percentile aggregations (`p50`, `p90`, `p95`, `p99`) as `quantile` | ||
| at the matching level. A chart config with `aggFn: 'p95'` rendered as the | ||
| nonexistent ClickHouse function `p95If`. The AI assistant now returns | ||
| percentiles with a level instead of a bare `quantile` (median), and v1 metric | ||
| chart conversion keeps the percentile level. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -23,6 +23,7 @@ import { | |
| convertToNumberChartConfig, | ||
| convertToTableChartConfig, | ||
| getAlignedDateRange, | ||
| getPercentileAggFnLevel, | ||
| Granularity, | ||
| hasPositiveSeriesLimit, | ||
| TIME_SERIES_DISPLAY_TYPE_BY_NAME, | ||
|
|
@@ -1111,16 +1112,11 @@ const mapV1AggFnToV2 = (aggFn?: AggFn): AggFnV2 | undefined => { | |
| } | ||
| // Map rate-based aggregations to their base aggregation | ||
| if (aggFn.endsWith('_rate')) { | ||
| return mapV1AggFnToV2(aggFn.replace('_rate', '') as AggFn); | ||
| return mapV1AggFnToV2(aggFn.replace(/_rate$/, '') as AggFn); | ||
| } | ||
|
|
||
| // Map percentiles to quantile | ||
| if ( | ||
| aggFn === 'p50' || | ||
| aggFn === 'p90' || | ||
| aggFn === 'p95' || | ||
| aggFn === 'p99' | ||
| ) { | ||
| if (getPercentileAggFnLevel(aggFn) != null) { | ||
| return 'quantile'; | ||
| } | ||
|
|
||
|
|
@@ -1218,8 +1214,12 @@ export const convertV1ChartConfigToV2 = ( | |
| const metricDataType = z | ||
| .nativeEnum(MetricsDataTypeV2) | ||
| .parse(rawMetricDataType?.toLowerCase()); | ||
| const percentileLevel = getPercentileAggFnLevel( | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 minor — v1 conversion works out the percentile level separately from the aggFn mapping
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 minor —
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 minor — The
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 minor — Percentile level is derived separately from
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 minor — The v1 conversion strips
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 minor — The
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond |
||
| s.aggFn?.replace(/_rate$/, ''), | ||
| ); | ||
| return { | ||
| aggFn: mapV1AggFnToV2(s.aggFn), | ||
| ...(percentileLevel != null ? { level: percentileLevel } : {}), | ||
| metricType: metricDataType, | ||
| valueExpression: field, | ||
| metricName, | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -3712,6 +3712,66 @@ describe('renderChartConfig', () => { | |
| expect(sql).toContain("hasToken(lower(Body), lower('denied'))"); | ||
| }); | ||
|
|
||
| describe('percentile shorthand aggFn', () => { | ||
| const baseTraceConfig: ChartConfigWithOptDateRange = { | ||
| displayType: DisplayType.Table, | ||
| connection: 'test-connection', | ||
| from: { | ||
| databaseName: 'default', | ||
| tableName: 'otel_traces', | ||
| }, | ||
| select: [], | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 minor — New percentile render tests never use a metric source, though that path depends on the rewrite running first Both new tests use a trace table. The rewrite is placed before the multi-series metric branch and Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond |
||
| where: '', | ||
| whereLanguage: 'sql', | ||
| timestampValueExpression: 'Timestamp', | ||
| dateRange: [new Date('2025-02-12'), new Date('2025-02-14')], | ||
| }; | ||
|
|
||
| it('renders p95 as quantile at level 0.95', async () => { | ||
| const config: ChartConfigWithOptDateRange = { | ||
| ...baseTraceConfig, | ||
| select: [ | ||
| { | ||
| aggFn: 'p95', | ||
| valueExpression: 'Duration', | ||
| aggCondition: "ServiceName = 'api'", | ||
| aggConditionLanguage: 'sql', | ||
| }, | ||
| ], | ||
| }; | ||
|
|
||
| const actual = parameterizedQueryToSql( | ||
| await renderChartConfig(config, mockMetadata, querySettings), | ||
| ); | ||
| expect(actual).toContain( | ||
| 'quantileIf(0.95)(toFloat64OrDefault(toString(Duration))', | ||
| ); | ||
| expect(actual).not.toContain('p95'); | ||
| }); | ||
|
|
||
| it('renders p99 as sample-weighted quantile at level 0.99', async () => { | ||
| const config: ChartConfigWithOptDateRange = { | ||
| ...baseTraceConfig, | ||
| sampleWeightExpression: 'SampleRate', | ||
| select: [ | ||
| { | ||
| aggFn: 'p99', | ||
| valueExpression: 'Duration', | ||
| aggCondition: '', | ||
| }, | ||
| ], | ||
| }; | ||
|
|
||
| const actual = parameterizedQueryToSql( | ||
| await renderChartConfig(config, mockMetadata, querySettings), | ||
| ); | ||
| expect(actual).toContain( | ||
| 'quantileTDigestWeighted(0.99)(toFloat64OrDefault(toString(Duration))', | ||
| ); | ||
| expect(actual).not.toContain('p99'); | ||
| }); | ||
| }); | ||
|
|
||
| describe('sample-weighted aggregations', () => { | ||
| const baseSampledConfig: ChartConfigWithOptDateRange = { | ||
| displayType: DisplayType.Table, | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -22,6 +22,7 @@ import { | |
| optimizeTimestampValueExpression, | ||
| parseToStartOfFunction, | ||
| pickBucketTimestampColumn, | ||
| resolvePercentileAggFn, | ||
| splitAndTrimWithBracket, | ||
| } from '@/core/utils'; | ||
| import { isPromqlChartConfig, isRawSqlChartConfig } from '@/guards'; | ||
|
|
@@ -3276,7 +3277,18 @@ export async function renderChartConfig( | |
|
|
||
| // Expand dashboard variables before anything reads the config's expressions, | ||
| // so metric translation and the CTEs it builds all see final SQL fragments. | ||
| const substitutedChartConfig = substituteChartConfigVariables(rawChartConfig); | ||
| const variableSubstitutedChartConfig = | ||
| substituteChartConfigVariables(rawChartConfig); | ||
| const substitutedChartConfig = Array.isArray( | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 minor — Percentiles are only resolved inside renderChartConfig, so the materialized-view optimizer still rejects saved
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 minor — Percentile shorthand is resolved only inside renderChartConfig, so the materialized-view optimizer still rejects
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 minor — Percentile rewrite runs only inside renderChartConfig, so the MV optimizer still gets the raw
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 minor — Percentile aggFns are only resolved inside renderChartConfig, so materialized-view optimization still rejects The app calls Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 minor — Percentile shorthand is resolved only inside the SQL renderer, so consumers that run before it still see
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond |
||
| variableSubstitutedChartConfig.select, | ||
| ) | ||
| ? { | ||
| ...variableSubstitutedChartConfig, | ||
| select: variableSubstitutedChartConfig.select.map( | ||
| resolvePercentileAggFn, | ||
| ), | ||
| } | ||
| : variableSubstitutedChartConfig; | ||
|
|
||
| // A metric chart with multiple series composes one query per series into a | ||
| // single UNION ALL + pivot statement (each metric type needs its own CTE | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🔵 minor — v1 conversion repeats the
_ratestripping and percentile check thatmapV1AggFnToV2already doesHave
mapV1AggFnToV2return{ aggFn, level }, or wrap its result withresolvePercentileAggFnon the_rate-stripped aggFn. Right now the call site strips_ratewith its own regex and computes the level separately from the function that decides the aggFn isquantile, so the two can disagree.Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond