Repository navigation
feat: PromQL Dashboards & Alerting #3114
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
2b0b9ea
a8378a0
75d3428
e7c35ab
9fae4d8
d31579d
de05c8d
fd99d36
cfb88f2
82f7780
5aac73f
48f6b81
5ce75bd
65e8baa
4f9fc9b
971e6c3
cd93902
4892a67
919136e
a184967
ca8c762
ca767ff
efa12b1
3593835
a7c09f5
0e10ff9
2fccafd
1ff7055
9847913
32462d1
c7a4108
d93d9c0
945ee15
c62072e
7a4d185
63c916d
0d523d8
3766926
414af4b
4c9b4f1
244ebe4
6aeb24d
1b7240b
04a4725
8fa8214
0e0f6f3
7ae01cb
50585fd
7668d38
72470f7
7ee681a
82220c5
aa27660
3eac1ee
3bcde7e
cbbb3cf
4654a9a
6522eae
3a22cec
faf2756
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| --- | ||
| '@hyperdx/api': minor | ||
| '@hyperdx/app': minor | ||
| '@hyperdx/common-utils': minor | ||
| --- | ||
|
|
||
| Added support for configuring and evaluating PromQL-based alerts directly from the chart explorer and dashboards. This allows triggering standard alert notifications based on the result of Prometheus queries. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -7,9 +7,12 @@ import z from 'zod'; | |
| import { ClickhouseClient } from '@/clickhouse'; | ||
| import { getConnectionById } from '@/controllers/connection'; | ||
| import { | ||
| connectionSupportsPrometheusHttpApi, | ||
| PROMETHEUS_MAX_EXECUTION_SEC, | ||
| PROMETHEUS_MAX_RESULT_ROWS, | ||
| CLICKHOUSE_PROMETHEUS_API_PREFIX, | ||
| clickhouseAuthHeaders, | ||
| clickhousePrometheusUpstream, | ||
| clickhouseServesPrometheusHttpApi, | ||
| joinPrometheusUpstreamUrl, | ||
| PROMETHEUS_CH_TIMEOUT_MS, | ||
| queryInstantViaTableFunction, | ||
| queryLabelNames, | ||
| queryLabelValues, | ||
|
|
@@ -21,6 +24,14 @@ import { getCounter, getHistogram } from '@/utils/instrumentation'; | |
| import logger from '@/utils/logger'; | ||
| import { objectIdSchema, stringListQueryParam } from '@/utils/zod'; | ||
|
|
||
| // Re-export helpers that were moved to timeseriesEngine; existing imports from | ||
|
Aryainguz marked this conversation as resolved.
Resolved
|
||
| // this module (e.g. integration tests) continue to work unchanged. | ||
| export { | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 minor — Production router re-exports helpers only for tests, and the move dropped their rationale Point prometheus.test.ts and prometheus.int.test.ts at Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond |
||
| CLICKHOUSE_PROMETHEUS_API_PREFIX, | ||
| clickhousePrometheusUpstream, | ||
| joinPrometheusUpstreamUrl, | ||
| } from '@/controllers/timeseriesEngine'; | ||
|
|
||
| const router = express.Router(); | ||
|
|
||
| // The proxy handlers catch their own errors and return Prometheus-shaped 4xx | ||
|
|
@@ -108,7 +119,6 @@ function getParams(req: express.Request): Record<string, string> { | |
| // -------------------------- | ||
|
|
||
| const PROMETHEUS_PROXY_TIMEOUT_MS = 90_000; | ||
| const PROMETHEUS_CH_TIMEOUT_MS = 30_000; | ||
| const PROMETHEUS_MAX_RESOLUTION = 11_000; | ||
| // Widest window /query_exemplars will proxy. Prometheus's exemplar store is a | ||
| // small circular buffer, so a wider range mostly costs a bigger streamed body | ||
|
|
@@ -118,13 +128,6 @@ export const PROMETHEUS_MAX_EXEMPLAR_WINDOW_SEC = 7 * 24 * 60 * 60; | |
| /** | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 minor — Unrelated deletion of the rationale behind The removed paragraph explained why Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 minor — Moving the helpers deleted their reasoning comments, including a security one The move removed the explanation of why Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond |
||
| * Whether a rejection from streaming the upstream body means the client hung up | ||
| * rather than the backend failing. | ||
| * | ||
| * The error code is the only usable signal. `pipeline` destroys every stream it | ||
| * touches before rejecting — the destination included, whichever end actually | ||
| * failed — so `res.destroyed` is true either way and testing it would classify | ||
| * every upstream fault as a user cancellation. Node also resolves the race in | ||
| * our favour: a real error always beats the ERR_STREAM_PREMATURE_CLOSE that the | ||
| * cascading destroy raises, never the other way round. | ||
| */ | ||
| export function isClientDisconnect(err: unknown): boolean { | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 minor — Moving the helpers deleted the comments explaining why the code is written this way These comments were dropped:
Restore these doc comments in timeseriesEngine.ts and prometheus.ts. Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond |
||
| return ( | ||
|
|
@@ -133,41 +136,6 @@ export function isClientDisconnect(err: unknown): boolean { | |
| ); | ||
| } | ||
|
|
||
| /** | ||
| * Join a Connection host with an absolute Prometheus API path. | ||
| * | ||
| * `new URL('/api/v1/query_range', 'http://host:8481/select/0/prometheus')` | ||
| * discards `/select/0/prometheus` because an absolute path replaces the base | ||
| * pathname. VictoriaMetrics cluster (and any Prometheus-compatible server | ||
| * mounted under a prefix) needs that prefix kept. Host userinfo, query, and | ||
| * hash are left untouched. | ||
| * | ||
| * `path` must be an absolute path (every call site passes a literal starting | ||
| * with `/`) -- this is not a general-purpose URL joiner. | ||
| * | ||
| * @see https://github.com/hyperdxio/hyperdx/issues/3046 | ||
| */ | ||
| export function joinPrometheusUpstreamUrl( | ||
| upstreamHost: string, | ||
| path: string, | ||
| ): URL { | ||
| const url = new URL(upstreamHost); | ||
| // `new URL('prometheus:9090')` succeeds with an opaque path (`prometheus:` | ||
| // scheme). The pathname setter is a no-op there, so without this guard the | ||
| // helper would return the host unchanged, `fetch` would fail, and the proxy | ||
| // would 502 / increment query_errors for a user misconfiguration. Same check | ||
| // as clickhouseProxy.ts. | ||
| if (url.protocol !== 'http:' && url.protocol !== 'https:') { | ||
| throw new TypeError('Connection host must be http(s)'); | ||
| } | ||
| // Strip ALL trailing slashes, not just one -- a host saved with a doubled | ||
| // trailing slash (e.g. `http://prom:9090//`) would otherwise leave a `//` | ||
| // in the joined path, which most servers treat as a distinct (404) path. | ||
| const basePath = url.pathname.replace(/\/+$/, ''); | ||
| url.pathname = `${basePath}${path}`; | ||
| return url; | ||
| } | ||
|
|
||
| // Only real Prometheus API params are ever caller-settable in | ||
| // proxyToPrometheus's query merge below. `params` there is built upstream by | ||
| // spreading the *entire* `req.query`/`req.body` with no allowlist (see | ||
|
|
@@ -192,32 +160,6 @@ const CALLER_SETTABLE_PARAM_KEYS = new Set([ | |
| 'table', | ||
| ]); | ||
|
|
||
| // ClickHouse serves the Prometheus HTTP API under this prefix on its main HTTP | ||
| // port when the `prometheus_api_v1` handler is configured (26.6+). | ||
| export const CLICKHOUSE_PROMETHEUS_API_PREFIX = '/prometheus/api/v1'; | ||
|
|
||
| /** | ||
| * The ClickHouse connection host with the query limits pinned as ClickHouse | ||
| * HTTP settings. The prometheus_api_v1 handler honours settings given in the | ||
| * URL and reports a breach as a Prometheus 400 (pinned by the live int test), | ||
| * so this is the only place to bound a PromQL evaluation — the request carries | ||
| * no SQL to attach SETTINGS to. Pinned on the host rather than passed as params | ||
| * so the merge in proxyToPrometheus (which lets the caller override any | ||
| * allowlisted key) can't loosen them. | ||
| */ | ||
| export function clickhousePrometheusUpstream( | ||
| host: string, | ||
| { | ||
| maxExecutionSec = PROMETHEUS_MAX_EXECUTION_SEC, | ||
| maxResultRows = PROMETHEUS_MAX_RESULT_ROWS, | ||
| } = {}, | ||
| ): string { | ||
| const url = new URL(host); | ||
| url.searchParams.set('max_execution_time', String(maxExecutionSec)); | ||
| url.searchParams.set('max_result_rows', String(maxResultRows)); | ||
| return url.toString(); | ||
| } | ||
|
|
||
| function newClickhouseClient(connection: { | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 minor — Security and behaviour rationale comments deleted during the move The move dropped the explanations for Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond |
||
| host: string; | ||
| username: string; | ||
|
|
@@ -231,59 +173,6 @@ function newClickhouseClient(connection: { | |
| }); | ||
| } | ||
|
|
||
| /** | ||
| * Whether a ClickHouse connection answers PromQL over HTTP. Two conditions, | ||
| * cheapest first: the server is new enough (26.6+), and the prometheus_api_v1 | ||
| * handler is actually configured — a self-managed server needs an | ||
| * `<http_handlers>` rule for that, which a stock install lacks. The handler | ||
| * answers every request, error or not, with Prometheus JSON (`{status, ...}`); | ||
| * a server without it answers a plain-text 404. `format_query` is used as the | ||
| * probe because it is the one endpoint that needs no table. | ||
| * | ||
| * Probed on every request, like the version check, so a handler enabled or | ||
| * removed on the server is reflected immediately. | ||
| */ | ||
| const PROBE_TIMEOUT_MS = 5_000; | ||
|
|
||
| async function clickhouseServesPrometheusHttpApi( | ||
| client: ClickhouseClient, | ||
| connection: { id: string; host: string; username: string; password?: string }, | ||
| ): Promise<boolean> { | ||
| if ( | ||
| !(await connectionSupportsPrometheusHttpApi({ | ||
| client, | ||
| connectionId: connection.id, | ||
| })) | ||
| ) { | ||
| return false; | ||
| } | ||
|
|
||
| try { | ||
| const url = joinPrometheusUpstreamUrl( | ||
| connection.host, | ||
| `${CLICKHOUSE_PROMETHEUS_API_PREFIX}/format_query`, | ||
| ); | ||
| url.searchParams.set('query', 'up'); | ||
| const resp = await fetch(url, { | ||
| headers: clickhouseAuthHeaders(connection), | ||
| signal: AbortSignal.timeout(PROBE_TIMEOUT_MS), | ||
| }); | ||
| return typeof JSON.parse(await resp.text())?.status === 'string'; | ||
| } catch { | ||
| return false; | ||
| } | ||
| } | ||
|
|
||
| function clickhouseAuthHeaders(connection: { | ||
| username: string; | ||
| password?: string; | ||
| }): Record<string, string> { | ||
| return { | ||
| 'X-ClickHouse-User': connection.username, | ||
| 'X-ClickHouse-Key': connection.password ?? '', | ||
| }; | ||
| } | ||
|
|
||
| // Forwards the response straight from the upstream Prometheus to the | ||
| // HyperDX client. Returns the HTTP status it wrote, so callers can record an | ||
| // error metric: this helper handles its own failures by writing 400/502/504 and | ||
|
|
||
Uh oh!
There was an error while loading. Please reload this page.