Repository navigation
fix: support https for the PromQL remote-write and scrape endpoint #3330
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
1cae7b0
c22f6dc
b705374
c753809
fdc7dfd
391a09d
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| --- | ||
| '@hyperdx/otel-collector': patch | ||
| '@hyperdx/api': patch | ||
| --- | ||
|
|
||
| fix: support https for the PromQL remote-write endpoint and ClickHouse scraping | ||
|
|
||
| `CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT` now accepts an `http://` or `https://` URL as well as a bare `host:port`. With `https://`, the collector's `prometheusremotewrite` exporter sends metrics over TLS and verifies the server certificate against the system CAs. Previously the exporter always used plain HTTP, so an HTTPS-only endpoint could not receive metrics. This applies to both standalone and OpAMP-managed collectors. | ||
|
|
||
| A bare `host:port` keeps plain HTTP, so existing setups are unchanged. The value must not include a path (`/write` is still appended automatically); the collector now exits at startup with a clear error if it does, or if the scheme is anything other than `http` or `https`. | ||
|
|
||
| The OpAMP-managed collector scraped the endpoint over plain HTTP regardless of its scheme. ClickHouse is now scraped in its own `clickhouse` job (previously the `otelcol` job) using the endpoint's scheme, with TLS verification on. The collector's own metrics endpoint stays on plain HTTP. A bare `host:port` or `http://` endpoint behaves as before. Scraped ClickHouse metrics now carry `job="clickhouse"` instead of `job="otelcol"`. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -24,6 +24,43 @@ if [ "$HYPERDX_OTEL_EXPORTER_CLICKHOUSE_JSON_ENABLE" = "true" ]; then | |
| export HYPERDX_OTEL_EXPORTER_CREATE_LEGACY_SCHEMA=true | ||
| fi | ||
|
|
||
| # CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT accepts a bare host:port (plain HTTP, | ||
| # the original format) or an http:// or https:// URL. Both the standalone | ||
| # config and the OpAMP-managed config (opampController.ts) need it as | ||
| # host:port, because it is also a Prometheus scrape target, so the scheme is | ||
| # split off here and handed to the prometheusremotewrite exporter through | ||
| # HYPERDX_PROMETHEUS_SCHEME and HYPERDX_PROMETHEUS_TLS_INSECURE. Runs before | ||
| # migrations so a bad value fails fast without touching ClickHouse. | ||
| prometheus_endpoint_error() { | ||
| echo "ERROR: CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT $1. Use host:port, http://host:port or https://host:port." >&2 | ||
| exit 1 | ||
| } | ||
|
|
||
| if [ -n "$CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT" ]; then | ||
| HYPERDX_PROMETHEUS_SCHEME=http | ||
| HYPERDX_PROMETHEUS_TLS_INSECURE=true | ||
| case "$CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT" in | ||
| https://*) | ||
| HYPERDX_PROMETHEUS_SCHEME=https | ||
| HYPERDX_PROMETHEUS_TLS_INSECURE=false | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 minor — Consider dropping Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond |
||
| ;; | ||
| http://*) ;; | ||
| # Only the scheme is echoed back: the rest of the URL may carry credentials. | ||
| *://*) prometheus_endpoint_error "has an unsupported scheme '${CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT%%://*}'" ;; | ||
| esac | ||
|
|
||
| PROMETHEUS_HOST_PORT="${CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT#*://}" | ||
| PROMETHEUS_HOST_PORT="${PROMETHEUS_HOST_PORT%/}" | ||
| case "$PROMETHEUS_HOST_PORT" in | ||
| "") prometheus_endpoint_error "has no host" ;; | ||
| */*) prometheus_endpoint_error "must not include a path (/write is appended automatically)" ;; | ||
| esac | ||
|
karl-power marked this conversation as resolved.
Resolved
|
||
|
|
||
| export CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT="$PROMETHEUS_HOST_PORT" | ||
| export HYPERDX_PROMETHEUS_SCHEME HYPERDX_PROMETHEUS_TLS_INSECURE | ||
| echo "CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT scheme: $HYPERDX_PROMETHEUS_SCHEME" | ||
| fi | ||
|
|
||
| # Run ClickHouse schema migrations if not using legacy schema creation | ||
| if [ "$HYPERDX_OTEL_EXPORTER_CREATE_LEGACY_SCHEMA" != "true" ]; then | ||
| # Run Go-based migrate tool with TLS support | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,40 @@ | ||
| { | ||
| "resourceMetrics": [ | ||
| { | ||
| "resource": { | ||
| "attributes": [ | ||
| { | ||
| "key": "suite-id", | ||
| "value": { | ||
| "stringValue": "promql-remote-write" | ||
| } | ||
| }, | ||
| { | ||
| "key": "service.name", | ||
| "value": { | ||
| "stringValue": "promql-remote-write-test-service" | ||
| } | ||
| } | ||
| ] | ||
| }, | ||
| "scopeMetrics": [ | ||
| { | ||
| "scope": {}, | ||
| "metrics": [ | ||
| { | ||
| "name": "smoke.promql.url.gauge", | ||
| "gauge": { | ||
| "dataPoints": [ | ||
| { | ||
| "timeUnixNano": "1901999590000000000", | ||
| "asDouble": 42 | ||
| } | ||
| ] | ||
| } | ||
| } | ||
| ] | ||
| } | ||
| ] | ||
| } | ||
| ] | ||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| #!/usr/bin/env bats | ||
|
|
||
| # HDX-4633: CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT may be a URL. entrypoint.sh | ||
| # splits the scheme off for the prometheusremotewrite exporter | ||
| # (config.standalone.promql.yaml) and keeps host:port for everything else. | ||
|
|
||
| load 'test_helpers/utilities.bash' | ||
| load 'test_helpers/assertions.bash' | ||
|
|
||
| @test "PromQL: an http:// endpoint URL still remote-writes metrics into the TimeSeries table" { | ||
|
karl-power marked this conversation as resolved.
Resolved
|
||
| run docker compose logs otel-collector-promql-url | ||
| [ "$status" -eq 0 ] | ||
| [[ "$output" == *"CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT scheme: http"* ]] | ||
|
karl-power marked this conversation as resolved.
Resolved
|
||
|
|
||
| # ClickHouse's remote_write handler (docker/clickhouse/local/config.xml) | ||
| # always writes to default.metrics_ts, but the collector only migrates that | ||
| # table with ENABLE_PROMQL=true and into its own database. Creating it here | ||
| # avoids racing the otel-collector service's migrations on `default`. | ||
| run clickhouse-client --port=9000 --query="CREATE TABLE IF NOT EXISTS default.metrics_ts ENGINE = TimeSeries SETTINGS allow_experimental_time_series_table = 1" | ||
| [ "$status" -eq 0 ] | ||
|
|
||
| emit_otel_data "http://localhost:64318" "data/metrics/promql-remote-write" "metrics" | ||
| wait_for_rows 9000 "SELECT count() FROM timeSeriesTags(default, metrics_ts) WHERE metric_name = 'smoke_promql_url_gauge' SETTINGS allow_experimental_time_series_table = 1" 1 | ||
| } | ||
|
|
||
| @test "PromQL: collector fails fast with a clear error when the endpoint includes a path" { | ||
| run docker compose logs otel-collector-promql-bad-endpoint | ||
| [ "$status" -eq 0 ] | ||
| [[ "$output" == *"CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT must not include a path"* ]] | ||
|
|
||
| run docker compose ps --status running --services | ||
| [ "$status" -eq 0 ] | ||
| [[ "$output" != *"otel-collector-promql-bad-endpoint"* ]] | ||
| } | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🔵 minor — With
https://there is no way to trust a private CA or skip verificationThe exporter only verifies against the system CAs, so a ClickHouse endpoint using a self-signed or internal-CA certificate (common for ClickHouse with TLS) still cannot receive metrics. The collector already reads
CLICKHOUSE_TLS_CA_FILEandCLICKHOUSE_TLS_INSECURE_SKIP_VERIFYfor the migrate tool (packages/otel-collector/cmd/migrate/main.go:71-75). Map those intotls.ca_file/tls.insecure_skip_verifyin both exporter configs, or document that only publicly trusted certificates work.Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond