Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions .changeset/promql-remote-write-https.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
---
'@hyperdx/otel-collector': patch
'@hyperdx/api': patch
---

fix: support https for the PromQL remote-write endpoint and ClickHouse scraping

`CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT` now accepts an `http://` or `https://` URL as well as a bare `host:port`. With `https://`, the collector's `prometheusremotewrite` exporter sends metrics over TLS and verifies the server certificate against the system CAs. Previously the exporter always used plain HTTP, so an HTTPS-only endpoint could not receive metrics. This applies to both standalone and OpAMP-managed collectors.

A bare `host:port` keeps plain HTTP, so existing setups are unchanged. The value must not include a path (`/write` is still appended automatically); the collector now exits at startup with a clear error if it does, or if the scheme is anything other than `http` or `https`.

The OpAMP-managed collector scraped the endpoint over plain HTTP regardless of its scheme. ClickHouse is now scraped in its own `clickhouse` job (previously the `otelcol` job) using the endpoint's scheme, with TLS verification on. The collector's own metrics endpoint stays on plain HTTP. A bare `host:port` or `http://` endpoint behaves as before. Scraped ClickHouse metrics now carry `job="clickhouse"` instead of `job="otelcol"`.
1 change: 1 addition & 0 deletions docker-compose.dev.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ services:
OTEL_COLLECTOR_CORE_VERSION: ${OTEL_COLLECTOR_CORE_VERSION}
environment:
CLICKHOUSE_ENDPOINT: 'tcp://ch-server:9000?dial_timeout=10s'
# host:port (plain HTTP), http://host:port or https://host:port
CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT: 'ch-server:9363'
HYPERDX_OTEL_EXPORTER_CLICKHOUSE_DATABASE: ${HYPERDX_OTEL_EXPORTER_CLICKHOUSE_DATABASE}
HYPERDX_API_KEY: ${HYPERDX_API_KEY}
Expand Down
9 changes: 7 additions & 2 deletions docker/otel-collector/config.standalone.promql.yaml
Original file line number Diff line number Diff line change
@@ -1,16 +1,21 @@
# Enables remote-write of OTLP metrics to a Prometheus-compatible endpoint.
# Only included when CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT is set.
#
# entrypoint.sh strips any http:// or https:// scheme from
# CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT and exports it as
# HYPERDX_PROMETHEUS_SCHEME / HYPERDX_PROMETHEUS_TLS_INSECURE. The defaults
# keep plain HTTP if those are unset.
#
# Mirrors the OpAMP-managed behavior in
# packages/api/src/opamp/controllers/opampController.ts (gated on
# IS_PROMQL_ENABLED). When updating this file, keep it in sync with
# buildOtelCollectorConfig().

exporters:
prometheusremotewrite:
endpoint: http://${env:CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT}/write
endpoint: ${env:HYPERDX_PROMETHEUS_SCHEME:-http}://${env:CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT}/write
tls:
insecure: true
insecure: ${env:HYPERDX_PROMETHEUS_TLS_INSECURE:-true}
resource_to_telemetry_conversion:
enabled: true

Expand Down
37 changes: 37 additions & 0 deletions docker/otel-collector/entrypoint.sh
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,43 @@ if [ "$HYPERDX_OTEL_EXPORTER_CLICKHOUSE_JSON_ENABLE" = "true" ]; then
export HYPERDX_OTEL_EXPORTER_CREATE_LEGACY_SCHEMA=true
fi

# CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT accepts a bare host:port (plain HTTP,
# the original format) or an http:// or https:// URL. Both the standalone
# config and the OpAMP-managed config (opampController.ts) need it as
# host:port, because it is also a Prometheus scrape target, so the scheme is
# split off here and handed to the prometheusremotewrite exporter through
# HYPERDX_PROMETHEUS_SCHEME and HYPERDX_PROMETHEUS_TLS_INSECURE. Runs before
# migrations so a bad value fails fast without touching ClickHouse.
prometheus_endpoint_error() {
echo "ERROR: CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT $1. Use host:port, http://host:port or https://host:port." >&2
exit 1
}

if [ -n "$CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT" ]; then
HYPERDX_PROMETHEUS_SCHEME=http
HYPERDX_PROMETHEUS_TLS_INSECURE=true
case "$CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT" in
https://*)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — With https:// there is no way to trust a private CA or skip verification

The exporter only verifies against the system CAs, so a ClickHouse endpoint using a self-signed or internal-CA certificate (common for ClickHouse with TLS) still cannot receive metrics. The collector already reads CLICKHOUSE_TLS_CA_FILE and CLICKHOUSE_TLS_INSECURE_SKIP_VERIFY for the migrate tool (packages/otel-collector/cmd/migrate/main.go:71-75). Map those into tls.ca_file / tls.insecure_skip_verify in both exporter configs, or document that only publicly trusted certificates work.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

HYPERDX_PROMETHEUS_SCHEME=https
HYPERDX_PROMETHEUS_TLS_INSECURE=false

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — HYPERDX_PROMETHEUS_TLS_INSECURE probably has no effect on the HTTP exporter

Consider dropping HYPERDX_PROMETHEUS_TLS_INSECURE, the ${env:...} placeholder for insecure in config.standalone.promql.yaml and opampController.ts, and the insecure: boolean → string type change, and let the URL scheme decide. For an HTTP client, configtls LoadTLSConfig returns a nil TLS config when insecure: true and no CA is set. Go's transport then still does verified TLS for an https:// URL using the system CAs, so the scheme alone already controls TLS. Verify this against the pinned collector version before removing.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

;;
http://*) ;;
# Only the scheme is echoed back: the rest of the URL may carry credentials.
*://*) prometheus_endpoint_error "has an unsupported scheme '${CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT%%://*}'" ;;
esac

PROMETHEUS_HOST_PORT="${CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT#*://}"
PROMETHEUS_HOST_PORT="${PROMETHEUS_HOST_PORT%/}"
case "$PROMETHEUS_HOST_PORT" in
"") prometheus_endpoint_error "has no host" ;;
*/*) prometheus_endpoint_error "must not include a path (/write is appended automatically)" ;;
esac
Comment thread
karl-power marked this conversation as resolved.
Resolved

export CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT="$PROMETHEUS_HOST_PORT"
export HYPERDX_PROMETHEUS_SCHEME HYPERDX_PROMETHEUS_TLS_INSECURE
echo "CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT scheme: $HYPERDX_PROMETHEUS_SCHEME"
fi

# Run ClickHouse schema migrations if not using legacy schema creation
if [ "$HYPERDX_OTEL_EXPORTER_CREATE_LEGACY_SCHEMA" != "true" ]; then
# Run Go-based migrate tool with TLS support
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,52 @@ describe('opampController', () => {
});
});

describe('buildOtelCollectorConfig prometheusremotewrite exporter', () => {
it('takes the scheme and TLS mode from the collector environment, defaulting to plain HTTP', () => {
configState.IS_PROMQL_ENABLED = true;

const cfg = buildOtelCollectorConfig([
{ apiKey: 'k1', collectorAuthenticationEnforced: false },
]);

// The defaults matter: collector images older than this change don't
// export HYPERDX_PROMETHEUS_*, and must keep today's plain-HTTP config.
expect(cfg.exporters?.prometheusremotewrite).toMatchObject({
endpoint:
'${env:HYPERDX_PROMETHEUS_SCHEME:-http}://${env:CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT}/write',
tls: { insecure: '${env:HYPERDX_PROMETHEUS_TLS_INSECURE:-true}' },
});
});
});

describe('buildOtelCollectorConfig prometheus receiver', () => {
it('scrapes ClickHouse in its own job so the scheme follows the endpoint', () => {
configState.IS_PROMQL_ENABLED = true;

const cfg = buildOtelCollectorConfig([
{ apiKey: 'k1', collectorAuthenticationEnforced: false },
]);

const scrapeConfigs = cfg.receivers.prometheus?.config.scrape_configs;
// The collector's own endpoint is always plain HTTP: no scheme here.
expect(scrapeConfigs?.[0]).toEqual({
job_name: 'otelcol',
scrape_interval: '30s',
static_configs: [{ targets: ['0.0.0.0:8888'] }],
});
// The default keeps collector images that predate
// HYPERDX_PROMETHEUS_SCHEME on plain HTTP.
expect(scrapeConfigs?.[1]).toEqual({
job_name: 'clickhouse',
scrape_interval: '30s',
scheme: '${env:HYPERDX_PROMETHEUS_SCHEME:-http}',
static_configs: [
{ targets: ['${env:CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT}'] },
],
});
});
});

describe('buildOtelCollectorConfig otlp/hyperdx authentication', () => {
it('accepts bare and bearer-prefixed authorization values for each API key', () => {
const cfg = buildOtelCollectorConfig([
Expand Down
26 changes: 19 additions & 7 deletions packages/api/src/opamp/controllers/opampController.ts
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,7 @@ type CollectorConfig = {
scrape_configs: Array<{
job_name: string;
scrape_interval: string;
scheme?: string;
static_configs: Array<{
targets: string[];
}>;
Expand Down Expand Up @@ -149,7 +150,7 @@ type CollectorConfig = {
prometheusremotewrite?: {
endpoint: string;
tls: {
insecure: boolean;
insecure: string;
};
resource_to_telemetry_conversion: {
enabled: boolean;
Expand Down Expand Up @@ -216,12 +217,19 @@ export const buildOtelCollectorConfig = (
{
job_name: 'otelcol',
scrape_interval: '30s',
static_configs: [{ targets: ['0.0.0.0:8888'] }],
},
// Separate job because the collector's own endpoint is always
// plain HTTP, while ClickHouse may be served over https. The
// default keeps collector images that predate
// HYPERDX_PROMETHEUS_SCHEME on http.
{
job_name: 'clickhouse',
scrape_interval: '30s',
scheme: '${env:HYPERDX_PROMETHEUS_SCHEME:-http}',
static_configs: [
{
targets: [
'0.0.0.0:8888',
'${env:CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT}',
],
targets: ['${env:CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT}'],
},
],
},
Expand Down Expand Up @@ -336,10 +344,14 @@ export const buildOtelCollectorConfig = (
);

if (config.IS_PROMQL_ENABLED && otelCollectorConfig.exporters) {
// The collector's entrypoint.sh derives the scheme and TLS mode from
// CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT. The defaults keep plain HTTP
// for collector images that predate it.
otelCollectorConfig.exporters.prometheusremotewrite = {
endpoint: 'http://${env:CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT}/write',
endpoint:
'${env:HYPERDX_PROMETHEUS_SCHEME:-http}://${env:CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT}/write',
tls: {
insecure: true,
insecure: '${env:HYPERDX_PROMETHEUS_TLS_INSECURE:-true}',
},
resource_to_telemetry_conversion: {
enabled: true,
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
{
"resourceMetrics": [
{
"resource": {
"attributes": [
{
"key": "suite-id",
"value": {
"stringValue": "promql-remote-write"
}
},
{
"key": "service.name",
"value": {
"stringValue": "promql-remote-write-test-service"
}
}
]
},
"scopeMetrics": [
{
"scope": {},
"metrics": [
{
"name": "smoke.promql.url.gauge",
"gauge": {
"dataPoints": [
{
"timeUnixNano": "1901999590000000000",
"asDouble": 42
}
]
}
}
]
}
]
}
]
}
50 changes: 50 additions & 0 deletions smoke-tests/otel-collector/docker-compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,56 @@ services:
ch-server:
condition: service_healthy

# HDX-4633: CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT given as a URL rather than
# a bare host:port. entrypoint.sh splits the scheme off for the
# prometheusremotewrite exporter; see promql-remote-write.bats.
otel-collector-promql-url:
build:
context: ../..
dockerfile: docker/otel-collector/Dockerfile
target: dev
args:
OTEL_COLLECTOR_VERSION: ${OTEL_COLLECTOR_VERSION:-0.155.0}
OTEL_COLLECTOR_CORE_VERSION: ${OTEL_COLLECTOR_CORE_VERSION:-1.61.0}
environment:
- CLICKHOUSE_ENDPOINT=tcp://ch-server:9000?dial_timeout=10s
- CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT=http://ch-server:9363/
- CLICKHOUSE_USER=default
- CLICKHOUSE_PASSWORD=
- HYPERDX_OTEL_EXPORTER_CLICKHOUSE_DATABASE=promql_url
- HYPERDX_LOG_LEVEL=info
- HYPERDX_OTEL_BATCH_TIMEOUT=100ms
# OPAMP_SERVER_URL is intentionally not set to run in standalone mode
ports:
- 64318:4318 # OTLP http receiver
networks:
- internal
depends_on:
ch-server:
condition: service_healthy

# HDX-4633: a path in CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT can't be used as
# a scrape target, so entrypoint.sh rejects it before migrations. Expected to
# exit immediately; see promql-remote-write.bats. CLICKHOUSE_ENDPOINT is
# unreachable on purpose, as in otel-collector-oidc-missing-audience.
otel-collector-promql-bad-endpoint:
build:
context: ../..
dockerfile: docker/otel-collector/Dockerfile
target: dev
args:
OTEL_COLLECTOR_VERSION: ${OTEL_COLLECTOR_VERSION:-0.155.0}
OTEL_COLLECTOR_CORE_VERSION: ${OTEL_COLLECTOR_CORE_VERSION:-1.61.0}
environment:
- CLICKHOUSE_ENDPOINT=tcp://127.0.0.1:1/?dial_timeout=1s
- CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT=https://ch-server:9363/write
- CLICKHOUSE_USER=default
- CLICKHOUSE_PASSWORD=
- HYPERDX_OTEL_EXPORTER_CLICKHOUSE_DATABASE=promql_bad_endpoint
- HYPERDX_LOG_LEVEL=info
networks:
- internal

# Older ClickHouse (< 26.2) to test the compatibility schema (bloom filters
# instead of full text indexes)
ch-server-compat:
Expand Down
34 changes: 34 additions & 0 deletions smoke-tests/otel-collector/promql-remote-write.bats
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
#!/usr/bin/env bats

# HDX-4633: CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT may be a URL. entrypoint.sh
# splits the scheme off for the prometheusremotewrite exporter
# (config.standalone.promql.yaml) and keeps host:port for everything else.

load 'test_helpers/utilities.bash'
load 'test_helpers/assertions.bash'

@test "PromQL: an http:// endpoint URL still remote-writes metrics into the TimeSeries table" {
Comment thread
karl-power marked this conversation as resolved.
Resolved
run docker compose logs otel-collector-promql-url
[ "$status" -eq 0 ]
[[ "$output" == *"CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT scheme: http"* ]]
Comment thread
karl-power marked this conversation as resolved.
Resolved

# ClickHouse's remote_write handler (docker/clickhouse/local/config.xml)
# always writes to default.metrics_ts, but the collector only migrates that
# table with ENABLE_PROMQL=true and into its own database. Creating it here
# avoids racing the otel-collector service's migrations on `default`.
run clickhouse-client --port=9000 --query="CREATE TABLE IF NOT EXISTS default.metrics_ts ENGINE = TimeSeries SETTINGS allow_experimental_time_series_table = 1"
[ "$status" -eq 0 ]

emit_otel_data "http://localhost:64318" "data/metrics/promql-remote-write" "metrics"
wait_for_rows 9000 "SELECT count() FROM timeSeriesTags(default, metrics_ts) WHERE metric_name = 'smoke_promql_url_gauge' SETTINGS allow_experimental_time_series_table = 1" 1
}

@test "PromQL: collector fails fast with a clear error when the endpoint includes a path" {
run docker compose logs otel-collector-promql-bad-endpoint
[ "$status" -eq 0 ]
[[ "$output" == *"CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT must not include a path"* ]]

run docker compose ps --status running --services
[ "$status" -eq 0 ]
[[ "$output" != *"otel-collector-promql-bad-endpoint"* ]]
}
6 changes: 4 additions & 2 deletions smoke-tests/otel-collector/setup_suite.bash
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,10 @@ setup_suite() {
wait_for_ready "otel-collector-custom"
wait_for_ready "otel-collector-oidc"
wait_for_ready "otel-collector-bearer-auth"
# otel-collector-oidc-missing-audience is expected to exit immediately,
# not become ready -- checked directly in oidc-auth.bats instead.
wait_for_ready "otel-collector-promql-url"
# otel-collector-oidc-missing-audience and otel-collector-promql-bad-endpoint
# are expected to exit immediately, not become ready -- checked directly in
# oidc-auth.bats and promql-remote-write.bats instead.
}

teardown_suite() {
Expand Down
Loading