Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
2b0b9ea
feat: Support Alerting for PromQL Charts
Aryainguz Sep 11, 2026
a8378a0
feat: add support for PromQL-based alerts in chart explorer and dashb…
Aryainguz Sep 11, 2026
75d3428
feat: support multi-series metrics and grouping in PromQL alert evalu…
Aryainguz Sep 11, 2026
e7c35ab
fix(alerts): address PR review comments for PromQL alerting
Aryainguz Sep 11, 2026
9fae4d8
fix(promql-alerts): address PR review feedback
Aryainguz Sep 12, 2026
d31579d
feat(promql): end-to-end alerting engine integration, UI previews, an…
Aryainguz Sep 13, 2026
de05c8d
fix(promql): resolve greptile PR review issues
Aryainguz Sep 13, 2026
fd99d36
style: run prettier on EditTimeChartForm
Aryainguz Sep 13, 2026
cfb88f2
removed any type
Aryainguz Sep 13, 2026
82f7780
revert: restore original error logging in clickhouse utils
Aryainguz Sep 13, 2026
5aac73f
revert: restore displayTypeSupportsPromQLAlerts and fix PromQL alert …
Aryainguz Sep 13, 2026
48f6b81
fix: allow PromQL alerts on Number and StackedBar display types
Aryainguz Sep 13, 2026
5ce75bd
style(alerts): remove ASCII banner comments
Aryainguz Sep 13, 2026
65e8baa
refactor(alerts): replace 'as' cast with 'satisfies' for PromqlConfig…
Aryainguz Sep 13, 2026
4f9fc9b
fix(alerts): address greptile review comments for promql alerts
Aryainguz Sep 13, 2026
971e6c3
fix: correct JSX indentation in PromqlChartEditor alert button
Aryainguz Sep 14, 2026
cd93902
chore: merge main into feat/promql-tile-alerts, resolve conflict in c…
Aryainguz Sep 14, 2026
4892a67
fix(alerts): address review comments for PromQL alerts
Aryainguz Sep 15, 2026
919136e
fix(alerts): address review comments for PromQL alerts
Aryainguz Sep 15, 2026
a184967
Merge branch 'main' into feat/promql-tile-alerts
Aryainguz Sep 15, 2026
ca8c762
fix: Avoid crashing on unpopulated alert.team reference by using isPo…
Aryainguz Sep 16, 2026
ca767ff
Merge main and resolve conflicts
Aryainguz Sep 16, 2026
efa12b1
test: correctly type formatMatrixResponse fixtures
Aryainguz Sep 16, 2026
3593835
fix(alerts): preserve zero-value threshold handling for PromQL alerts
Aryainguz Sep 16, 2026
a7c09f5
lint
Aryainguz Sep 16, 2026
0e10ff9
test(alerts): fix PromQL mock timestamp to reflect Prometheus step ev…
Aryainguz Sep 16, 2026
2fccafd
test(alerts): update group key expectation for PromQL test
Aryainguz Sep 16, 2026
1ff7055
refactor: move timestamp and step parsing above prometheus endpoint p…
Aryainguz Sep 16, 2026
9847913
fix: lint
Aryainguz Sep 17, 2026
32462d1
Merge branch 'main' into feat/promql-tile-alerts
Aryainguz Sep 17, 2026
c7a4108
fix: skip Prometheus max resolution check for proxy endpoints and cap…
Aryainguz Sep 17, 2026
d93d9c0
Merge branch 'main' into feat/promql-tile-alerts
Aryainguz Sep 18, 2026
945ee15
chore: fix lint errors in utils.ts
Aryainguz Sep 18, 2026
c62072e
chore: remove unused useController import
Aryainguz Sep 18, 2026
7a4d185
chore: fix checkAlerts tests for queryPrometheusRangeFromClickHouse r…
Aryainguz Sep 23, 2026
63c916d
chore: Resolve merge conflicts and centralize PromQL backend routing
Aryainguz Sep 23, 2026
0d523d8
style: format checkAlerts index and test files
Aryainguz Sep 23, 2026
3766926
fix: lint errors (unused vars and duplicate imports)
Aryainguz Sep 23, 2026
414af4b
refactor: unexport prometheus constants and helper function in timese…
Aryainguz Sep 23, 2026
4c9b4f1
test: fix failing test expecting single Alert Name validation element
Aryainguz Sep 23, 2026
244ebe4
fix: address P1 defects in PromQL alerts (blank expressions & clickho…
Aryainguz Sep 23, 2026
6aeb24d
refactor: pass connection and clickhouse client to evaluatePromqlAler…
Aryainguz Sep 23, 2026
1b7240b
test: update checkAlerts task assertions and clean up lint comments
Aryainguz Sep 23, 2026
04a4725
Merge branch 'main' into feat/promql-tile-alerts
Aryainguz Sep 23, 2026
8fa8214
Merge branch 'main' into feat/promql-tile-alerts
Aryainguz Sep 25, 2026
0e0f6f3
fix: correct expected buckets for promql alerts
Aryainguz Sep 26, 2026
7ae01cb
Merge branch 'main' into feat/promql-tile-alerts
Aryainguz Sep 26, 2026
50585fd
fix(alerts): properly auto-resolve disappearing groups in PromQL and …
Aryainguz Sep 26, 2026
7668d38
fix(review): address open review comments
Aryainguz Sep 26, 2026
72470f7
lint
Aryainguz Sep 26, 2026
7ee681a
Merge branch 'main' into feat/promql-tile-alerts
Aryainguz Sep 29, 2026
82220c5
feat: address review comments, add PromQL alert tests, and remove unn…
Aryainguz Sep 29, 2026
aa27660
Merge branch 'main' into feat/promql-tile-alerts
Aryainguz Oct 2, 2026
3eac1ee
refactor(checkAlerts): consolidate duplicated ~250-line bucket evalua…
Aryainguz Oct 2, 2026
3bcde7e
refactor(checkAlerts): consolidate duplicated ~250-line bucket evalua…
Aryainguz Oct 2, 2026
cbbb3cf
lint
Aryainguz Oct 2, 2026
4654a9a
lint
Aryainguz Oct 2, 2026
6522eae
test: restore all jest mocks after each checkAlerts integration test
Aryainguz Oct 2, 2026
3a22cec
test: restore all jest mocks after each checkAlerts integration test
Aryainguz Oct 2, 2026
faf2756
feat: add support for PromQL alerts in default alert check provider
Aryainguz Oct 2, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .changeset/promql-alerts.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
'@hyperdx/api': minor
'@hyperdx/app': minor
'@hyperdx/common-utils': minor
---

Added support for configuring and evaluating PromQL-based alerts directly from the chart explorer and dashboards. This allows triggering standard alert notifications based on the result of Prometheus queries.
101 changes: 96 additions & 5 deletions packages/api/src/controllers/timeseriesEngine.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,8 @@ import {
import { ClickhouseClient } from '@/clickhouse';
import logger from '@/utils/logger';

export const PROMETHEUS_MAX_EXECUTION_SEC = 30;
export const PROMETHEUS_MAX_RESULT_ROWS = 100000;
const PROMETHEUS_MAX_EXECUTION_SEC = 30;
const PROMETHEUS_MAX_RESULT_ROWS = 100000;

export type TimeSeriesTagsQueryArgs = {
client: ClickhouseClient;
Expand Down Expand Up @@ -254,9 +254,7 @@ export async function queryLabelNames({
return queryDistinctTagsValues({ ...args, value, conditions, limit });
}

// --------------------------
// PromQL via table functions (ClickHouse < 26.6)
// --------------------------
//
// Servers without the `prometheus_api_v1` HTTP handler can still evaluate
// PromQL through `prometheusQuery`/`prometheusQueryRange`. ClickHouse only
Expand Down Expand Up @@ -394,7 +392,7 @@ export async function queryInstantViaTableFunction({
* server is picked up without a restart; an unknown version is treated as
* old, matching the other version-gated features.
*/
export async function connectionSupportsPrometheusHttpApi({
async function connectionSupportsPrometheusHttpApi({
client,
connectionId,
}: {
Expand All @@ -406,3 +404,96 @@ export async function connectionSupportsPrometheusHttpApi({
await metadata.getServerVersion({ connectionId }),
);
}

// Shared Prometheus connection helpers
Comment thread
Aryainguz marked this conversation as resolved.
Resolved
//
// Used by both the HTTP proxy (routers/api/prometheus.ts) and the alerting
// task (tasks/checkAlerts) so backend-selection logic stays in one place.

/** ClickHouse prefix for the prometheus_api_v1 HTTP handler (26.6+). */
export const CLICKHOUSE_PROMETHEUS_API_PREFIX = '/prometheus/api/v1';

/**
* Timeout for PromQL evaluation via ClickhouseClient.
* Shared with checkAlerts so both callers use the same limit.
*/
export const PROMETHEUS_CH_TIMEOUT_MS = 30_000;

const PROM_PROBE_TIMEOUT_MS = 5_000;

/**
* Join a Connection host with an absolute Prometheus API path, preserving any
* prefix already baked into the host (e.g. VictoriaMetrics cluster prefix).
*/
export function joinPrometheusUpstreamUrl(
upstreamHost: string,
path: string,
): URL {
const url = new URL(upstreamHost);
if (url.protocol !== 'http:' && url.protocol !== 'https:') {
throw new TypeError('Connection host must be http(s)');
}
const basePath = url.pathname.replace(/\/+$/, '');
url.pathname = `${basePath}${path}`;
return url;
}

/** Basic-auth headers for ClickHouse's native HTTP API. */
export function clickhouseAuthHeaders(connection: {
username: string;
password?: string;
}): Record<string, string> {
return {
'X-ClickHouse-User': connection.username,
'X-ClickHouse-Key': connection.password ?? '',
};
}

/**
* ClickHouse host URL with PromQL query-limits pinned as HTTP settings so
* the prometheus_api_v1 handler enforces them.
*/
export function clickhousePrometheusUpstream(
host: string,
{
maxExecutionSec = PROMETHEUS_MAX_EXECUTION_SEC,
maxResultRows = PROMETHEUS_MAX_RESULT_ROWS,
} = {},
): string {
const url = new URL(host);
url.searchParams.set('max_execution_time', String(maxExecutionSec));
url.searchParams.set('max_result_rows', String(maxResultRows));
return url.toString();
}

/**
* Whether this ClickHouse instance serves the Prometheus HTTP API.
* Probed on each call so a config change is reflected immediately.
*/
export async function clickhouseServesPrometheusHttpApi(
client: ClickhouseClient,
connection: { id: string; host: string; username: string; password?: string },
): Promise<boolean> {
if (
!(await connectionSupportsPrometheusHttpApi({
client,
connectionId: connection.id,
}))
) {
return false;
}
try {
const url = joinPrometheusUpstreamUrl(
connection.host,
`${CLICKHOUSE_PROMETHEUS_API_PREFIX}/format_query`,
);
url.searchParams.set('query', 'up');
const resp = await fetch(url, {
headers: clickhouseAuthHeaders(connection),
signal: AbortSignal.timeout(PROM_PROBE_TIMEOUT_MS),
});
return typeof JSON.parse(await resp.text())?.status === 'string';
} catch {
return false;
}
}
139 changes: 14 additions & 125 deletions packages/api/src/routers/api/prometheus.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,12 @@ import z from 'zod';
import { ClickhouseClient } from '@/clickhouse';
import { getConnectionById } from '@/controllers/connection';
import {
connectionSupportsPrometheusHttpApi,
PROMETHEUS_MAX_EXECUTION_SEC,
PROMETHEUS_MAX_RESULT_ROWS,
CLICKHOUSE_PROMETHEUS_API_PREFIX,
clickhouseAuthHeaders,
clickhousePrometheusUpstream,
clickhouseServesPrometheusHttpApi,
joinPrometheusUpstreamUrl,
PROMETHEUS_CH_TIMEOUT_MS,
queryInstantViaTableFunction,
queryLabelNames,
queryLabelValues,
Expand All @@ -21,6 +24,14 @@ import { getCounter, getHistogram } from '@/utils/instrumentation';
import logger from '@/utils/logger';
import { objectIdSchema, stringListQueryParam } from '@/utils/zod';

// Re-export helpers that were moved to timeseriesEngine; existing imports from
Comment thread
Aryainguz marked this conversation as resolved.
Resolved
// this module (e.g. integration tests) continue to work unchanged.
export {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Production router re-exports helpers only for tests, and the move dropped their rationale

Point prometheus.test.ts and prometheus.int.test.ts at @/controllers/timeseriesEngine and remove the re-export. Restore the removed doc comments on joinPrometheusUpstreamUrl (#3046 prefix handling, opaque-scheme guard), clickhousePrometheusUpstream (why limits are pinned on the host), and isClientDisconnect (why res.destroyed cannot be used). They explain non-obvious constraints.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

CLICKHOUSE_PROMETHEUS_API_PREFIX,
clickhousePrometheusUpstream,
joinPrometheusUpstreamUrl,
} from '@/controllers/timeseriesEngine';

const router = express.Router();

// The proxy handlers catch their own errors and return Prometheus-shaped 4xx
Expand Down Expand Up @@ -108,7 +119,6 @@ function getParams(req: express.Request): Record<string, string> {
// --------------------------

const PROMETHEUS_PROXY_TIMEOUT_MS = 90_000;
const PROMETHEUS_CH_TIMEOUT_MS = 30_000;
const PROMETHEUS_MAX_RESOLUTION = 11_000;
// Widest window /query_exemplars will proxy. Prometheus's exemplar store is a
// small circular buffer, so a wider range mostly costs a bigger streamed body
Expand All @@ -118,13 +128,6 @@ export const PROMETHEUS_MAX_EXEMPLAR_WINDOW_SEC = 7 * 24 * 60 * 60;
/**

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Unrelated deletion of the rationale behind isClientDisconnect, and the docs lost when the helpers moved

The removed paragraph explained why res.destroyed must not be checked. Moving joinPrometheusUpstreamUrl also dropped the opaque-scheme guard rationale and the #3046 link, and the moved clickhousePrometheusUpstream lost its note about why limits are pinned on the host. Restore these comments on the moved helpers in timeseriesEngine.ts and on isClientDisconnect.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Moving the helpers deleted their reasoning comments, including a security one

The move removed the explanation of why isClientDisconnect must not check res.destroyed, the VictoriaMetrics prefix and opaque-URL guard notes on joinPrometheusUpstreamUrl (#3046), and why limits are pinned on the host so the caller-param merge can't loosen them (clickhousePrometheusUpstream). Move those comments with the code into timeseriesEngine.ts. Also point the int tests at the new module instead of keeping the re-export shim.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

* Whether a rejection from streaming the upstream body means the client hung up
* rather than the backend failing.
*
* The error code is the only usable signal. `pipeline` destroys every stream it
* touches before rejecting — the destination included, whichever end actually
* failed — so `res.destroyed` is true either way and testing it would classify
* every upstream fault as a user cancellation. Node also resolves the race in
* our favour: a real error always beats the ERR_STREAM_PREMATURE_CLOSE that the
* cascading destroy raises, never the other way round.
*/
export function isClientDisconnect(err: unknown): boolean {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Moving the helpers deleted the comments explaining why the code is written this way

These comments were dropped:

Restore these doc comments in timeseriesEngine.ts and prometheus.ts.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

return (
Expand All @@ -133,41 +136,6 @@ export function isClientDisconnect(err: unknown): boolean {
);
}

/**
* Join a Connection host with an absolute Prometheus API path.
*
* `new URL('/api/v1/query_range', 'http://host:8481/select/0/prometheus')`
* discards `/select/0/prometheus` because an absolute path replaces the base
* pathname. VictoriaMetrics cluster (and any Prometheus-compatible server
* mounted under a prefix) needs that prefix kept. Host userinfo, query, and
* hash are left untouched.
*
* `path` must be an absolute path (every call site passes a literal starting
* with `/`) -- this is not a general-purpose URL joiner.
*
* @see https://github.com/hyperdxio/hyperdx/issues/3046
*/
export function joinPrometheusUpstreamUrl(
upstreamHost: string,
path: string,
): URL {
const url = new URL(upstreamHost);
// `new URL('prometheus:9090')` succeeds with an opaque path (`prometheus:`
// scheme). The pathname setter is a no-op there, so without this guard the
// helper would return the host unchanged, `fetch` would fail, and the proxy
// would 502 / increment query_errors for a user misconfiguration. Same check
// as clickhouseProxy.ts.
if (url.protocol !== 'http:' && url.protocol !== 'https:') {
throw new TypeError('Connection host must be http(s)');
}
// Strip ALL trailing slashes, not just one -- a host saved with a doubled
// trailing slash (e.g. `http://prom:9090//`) would otherwise leave a `//`
// in the joined path, which most servers treat as a distinct (404) path.
const basePath = url.pathname.replace(/\/+$/, '');
url.pathname = `${basePath}${path}`;
return url;
}

// Only real Prometheus API params are ever caller-settable in
// proxyToPrometheus's query merge below. `params` there is built upstream by
// spreading the *entire* `req.query`/`req.body` with no allowlist (see
Expand All @@ -192,32 +160,6 @@ const CALLER_SETTABLE_PARAM_KEYS = new Set([
'table',
]);

// ClickHouse serves the Prometheus HTTP API under this prefix on its main HTTP
// port when the `prometheus_api_v1` handler is configured (26.6+).
export const CLICKHOUSE_PROMETHEUS_API_PREFIX = '/prometheus/api/v1';

/**
* The ClickHouse connection host with the query limits pinned as ClickHouse
* HTTP settings. The prometheus_api_v1 handler honours settings given in the
* URL and reports a breach as a Prometheus 400 (pinned by the live int test),
* so this is the only place to bound a PromQL evaluation — the request carries
* no SQL to attach SETTINGS to. Pinned on the host rather than passed as params
* so the merge in proxyToPrometheus (which lets the caller override any
* allowlisted key) can't loosen them.
*/
export function clickhousePrometheusUpstream(
host: string,
{
maxExecutionSec = PROMETHEUS_MAX_EXECUTION_SEC,
maxResultRows = PROMETHEUS_MAX_RESULT_ROWS,
} = {},
): string {
const url = new URL(host);
url.searchParams.set('max_execution_time', String(maxExecutionSec));
url.searchParams.set('max_result_rows', String(maxResultRows));
return url.toString();
}

function newClickhouseClient(connection: {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Security and behaviour rationale comments deleted during the move

The move dropped the explanations for joinPrometheusUpstreamUrl (VictoriaMetrics prefix, #3046, opaque-URL guard), clickhousePrometheusUpstream (why limits are pinned on the host so callers can't loosen them), CALLER_SETTABLE_PARAM_KEYS (the extra_label tenant-scope bypass), and isClientDisconnect, even though the code is unchanged. Carry these doc comments over to timeseriesEngine.ts and restore the ones left in place.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

host: string;
username: string;
Expand All @@ -231,59 +173,6 @@ function newClickhouseClient(connection: {
});
}

/**
* Whether a ClickHouse connection answers PromQL over HTTP. Two conditions,
* cheapest first: the server is new enough (26.6+), and the prometheus_api_v1
* handler is actually configured — a self-managed server needs an
* `<http_handlers>` rule for that, which a stock install lacks. The handler
* answers every request, error or not, with Prometheus JSON (`{status, ...}`);
* a server without it answers a plain-text 404. `format_query` is used as the
* probe because it is the one endpoint that needs no table.
*
* Probed on every request, like the version check, so a handler enabled or
* removed on the server is reflected immediately.
*/
const PROBE_TIMEOUT_MS = 5_000;

async function clickhouseServesPrometheusHttpApi(
client: ClickhouseClient,
connection: { id: string; host: string; username: string; password?: string },
): Promise<boolean> {
if (
!(await connectionSupportsPrometheusHttpApi({
client,
connectionId: connection.id,
}))
) {
return false;
}

try {
const url = joinPrometheusUpstreamUrl(
connection.host,
`${CLICKHOUSE_PROMETHEUS_API_PREFIX}/format_query`,
);
url.searchParams.set('query', 'up');
const resp = await fetch(url, {
headers: clickhouseAuthHeaders(connection),
signal: AbortSignal.timeout(PROBE_TIMEOUT_MS),
});
return typeof JSON.parse(await resp.text())?.status === 'string';
} catch {
return false;
}
}

function clickhouseAuthHeaders(connection: {
username: string;
password?: string;
}): Record<string, string> {
return {
'X-ClickHouse-User': connection.username,
'X-ClickHouse-Key': connection.password ?? '',
};
}

// Forwards the response straight from the upstream Prometheus to the
// HyperDX client. Returns the HTTP status it wrote, so callers can record an
// error metric: this helper handles its own failures by writing 400/502/504 and
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -602,9 +602,11 @@ describe('internal schema field classification', () => {

it.each(
AlertChartConfigSchema.options.flatMap((member, i) =>
Object.keys(member.shape).map(
key => [i === 0 ? 'builder' : 'raw SQL', key] as const,
),
Object.keys(member.shape).map(key => {
const variantName =
i === 0 ? 'builder' : i === 1 ? 'raw SQL' : 'promql';
return [variantName, key] as const;
}),
),
)('classifies the internal %s config field "%s"', (_variant, key) => {
const classified =
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,10 @@ export const EVALUATION_INERT_CONFIG_KEYS: ReadonlySet<string> = new Set([
// granularity: 'auto' on every inline alert it authors — refusing on it
// would blank chartConfig for the most common authoring path.
'granularity',
// Render-only: only controls the legend label in the UI, has no effect on
// alert evaluation. Classifying as lossy would block external PUT for any
// alert carrying a legend.
'legendTemplate',
]);

/**
Expand All @@ -143,6 +147,8 @@ export const KNOWN_LOSSY_CONFIG_KEYS: ReadonlySet<string> = new Set([
'bodyExpression',
'useTextIndexForImplicitColumn',
'metricTables',
'promqlExpression',
'step',
]);

/** Select-item fields the external dialect round-trips. */
Expand Down
Loading
Loading