Skip to content

fix: support https for the PromQL remote-write and scrape endpoint - #3330

Merged
kodiakhq[bot] merged 6 commits into
mainfrom
allow-https-hosts-for-external-prom-ql-endpoints
Oct 19, 2026
Merged

kodiakhq[bot] merged 6 commits into
mainfrom
allow-https-hosts-for-external-prom-ql-endpoints

Conversation

@karl-power

@karl-power karl-power commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Why

The PromQL remote-write exporter was hardcoded to http:// with insecure: true, so an HTTPS-only endpoint couldn’t receive metrics. The exporter now takes its scheme from CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT, in both the standalone config and the OpAMP-managed config.

What changed

  • docker/otel-collector/entrypoint.sh accepts host:port, http://host:port or https://host:port. It strips the scheme, puts the bare host:port back into CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT (the value is also used as a scrape target, which can't take a scheme), and exports HYPERDX_PROMETHEUS_SCHEME and HYPERDX_PROMETHEUS_TLS_INSECURE.
  • https:// turns on TLS with certificate verification against the system CAs. http:// or no scheme keeps today's plain HTTP.
  • config.standalone.promql.yaml and opampController.ts build the exporter endpoint and TLS setting from those env vars. The fallbacks (:-http, :-true) keep collector images that predate this change on plain HTTP.
  • A value with a path (e.g. /write), an unsupported scheme or no host now fails at startup with a clear error, before migrations run.
  • The OpAMP config scrapes ClickHouse in its own clickhouse job with scheme: ${env:HYPERDX_PROMETHEUS_SCHEME:-http}. The collector's own 0.0.0.0:8888 target stays in the otelcol job, since it is always plain HTTP. TLS verification stays on. Scraped ClickHouse metrics now carry job="clickhouse" (and service.name="clickhouse") instead of otelcol. Nothing in the repo filters on the old value.
  • Tests: unit tests for the OpAMP exporter and scrape config, and smoke tests for an http:// URL and for the path error.

How to test

1. Startup script parsing (no Docker)

From the repo root, define a helper that runs just the endpoint-parsing block of entrypoint.sh:

check() { { sed -n '/^prometheus_endpoint_error()/,/^fi$/p' docker/otel-collector/entrypoint.sh; echo 'env | grep PROMETHEUS'; } | CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT="$1" sh; }

Then run each case:

check ch-server:9363
check http://ch-server:9363/
check https://ch.example.com:9363
check ftp://x:1
check https://x:9363/write
check https://

Expected:

  • ch-server:9363 → http, insecure true, endpoint unchanged
  • http://ch-server:9363/ → http, true, endpoint ch-server:9363
  • https://ch.example.com:9363 → https, false, endpoint ch.example.com:9363
  • ftp://x:1, https://x:9363/write, https:// → each prints an error

2. Smoke tests (Docker + brew install bats-core)

cd smoke-tests/otel-collector && bats *.bats

3. Dev stack, OpAMP path

Start the dev stack with yarn dev. Find the container names with docker ps | grep -E 'otel-collector|ch-server'.

In docker-compose.dev.yml, set CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT on the otel-collector service (not otel-collector-json) to each value below. Stop and start the stack after each change, then wait ~2 minutes.

Remote-write and scrape logs:

docker logs --since 2m <otel-collector-container> 2>&1 | grep -v TransformContext | grep -oE 'scheme: https?|remote write request successful|https?://[a-z0-9.-]+:[0-9]+/(write|metrics)|HTTP response to HTTPS client|scrape_pool[^,]*clickhouse' | sort | uniq -c

ClickHouse data (run twice a minute apart; latest should advance):

docker exec <ch-server-container> clickhouse-client --query "SELECT count() FROM timeSeriesTags(default, metrics_ts) SETTINGS allow_experimental_time_series_table = 1"
docker exec <ch-server-container> clickhouse-client --query "SELECT ResourceAttributes['service.name'] AS svc, count() AS n, max(TimeUnix) AS latest FROM default.otel_metrics_gauge WHERE TimeUnix > now() - INTERVAL 1 MINUTE AND MetricName LIKE 'ClickHouse%' GROUP BY svc"

Expected:

  • ch-server:9363 (default): scheme: http, http://ch-server:9363/write, remote write request successful, no Scrape failed. A clickhouse row with a recent latest, and a rising time series count.
  • http://ch-server:9363: same as the default.
  • https://ch-server:9363: scheme: https, https://ch-server:9363/write and HTTP response to HTTPS client for remote-write. The scrape also fails with Scrape failed for scrape_pool: clickhouse, target https://ch-server:9363/metrics. Both errors are the expected result: local ClickHouse only serves plain HTTP, so they show the collector now uses HTTPS for both. The otelcol job still scrapes fine.

Revert docker-compose.dev.yml afterwards.

Linear

Closes HDX-4633

@changeset-bot

changeset-bot Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 391a09d

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages
Name Type
@hyperdx/otel-collector Patch
@hyperdx/api Patch
@hyperdx/app Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercel Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Oct 19, 2026 12:00pm UTC
hyperdx-storybook Ready Ready Preview Oct 19, 2026 12:00pm UTC

Request Review

@karl-power
karl-power marked this pull request as draft October 8, 2026 16:38
@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Duplicate code analysis

No new duplicate code introduced by this PR.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 125 in total (1.69%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

case "$CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT" in
https://*)
HYPERDX_PROMETHEUS_SCHEME=https
HYPERDX_PROMETHEUS_TLS_INSECURE=false

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — HYPERDX_PROMETHEUS_TLS_INSECURE probably has no effect on the HTTP exporter

Consider dropping HYPERDX_PROMETHEUS_TLS_INSECURE, the ${env:...} placeholder for insecure in config.standalone.promql.yaml and opampController.ts, and the insecure: boolean → string type change, and let the URL scheme decide. For an HTTP client, configtls LoadTLSConfig returns a nil TLS config when insecure: true and no CA is set. Go's transport then still does verified TLS for an https:// URL using the system CAs, so the scheme alone already controls TLS. Verify this against the pinned collector version before removing.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

HYPERDX_PROMETHEUS_SCHEME=http
HYPERDX_PROMETHEUS_TLS_INSECURE=true
case "$CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT" in
https://*)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — With https:// there is no way to trust a private CA or skip verification

The exporter only verifies against the system CAs, so a ClickHouse endpoint using a self-signed or internal-CA certificate (common for ClickHouse with TLS) still cannot receive metrics. The collector already reads CLICKHOUSE_TLS_CA_FILE and CLICKHOUSE_TLS_INSECURE_SKIP_VERIFY for the migrate tool (packages/otel-collector/cmd/migrate/main.go:71-75). Map those into tls.ca_file / tls.insecure_skip_verify in both exporter configs, or document that only publicly trusted certificates work.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 critical or 🟠 major in code this PR adds or changes: fix before asking for review.
  • 🔵 minor in code this PR changes: your call. Fix if small, otherwise reply.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    severity. Reply with one sentence; if it is critical, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

1 finding(s): 🔴 0 critical · 🟠 0 major · 🔵 1 minor

1 posted as inline comment(s) on the changed lines.


Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides.

@greptile-apps

greptile-apps Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; no new actionable defects were found.

Summary

Adds HTTPS support for PromQL remote writes and OpAMP-managed ClickHouse scraping.

  • PromQL remote writes use the scheme supplied with the endpoint.
  • OpAMP scrapes ClickHouse with its own endpoint scheme.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Configured endpoint] --> B[Entrypoint separates scheme and address]
  B --> C[Standalone remote-write config]
  B --> D[OpAMP remote-write config]
  B --> E[OpAMP ClickHouse scrape job]
  C --> F[HTTP or verified HTTPS]
  D --> F
  E --> F
  G[Collector self-scrape] --> H[Plain HTTP]
Loading

Reviews (5) · Last reviewed commit: "Merge branch 'main' into allow-https-hos..." · Reviewed by Greptile

Comment thread docker/otel-collector/entrypoint.sh Resolved
@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

✅ No critical issues found.

No P0/P1 ship-blockers: the change is backward-compatible for bare host:port and http:// endpoints (defaults preserve plain HTTP), fails fast on malformed input before migrations, and the OpAMP env propagation was verified correct for matched collector/API versions. The findings below are recommended hardening and nitpicks.

🟡 P2 -- recommended

  • packages/api/src/opamp/controllers/opampController.ts:227 -- Splitting the ClickHouse scrape target into its own clickhouse job changes the job label on all scraped ClickHouse series from otelcol to clickhouse, silently breaking any operator dashboard or alert that filters on job="otelcol".
    • Fix: Document the job label change in the changeset/upgrade notes, or preserve job="otelcol" for the ClickHouse target via relabeling.
    • adversarial-reviewer, reliability-reviewer, kieran-typescript-reviewer
  • docker/otel-collector/entrypoint.sh:45 -- An https:// endpoint hard-sets HYPERDX_PROMETHEUS_TLS_INSECURE=false and the clickhouse scrape job carries no tls_config, so a ClickHouse served over HTTPS with a self-signed or private-CA certificate fails both remote-write and scrape with no override.
    • Fix: Honor a pre-set HYPERDX_PROMETHEUS_TLS_INSECURE rather than clobbering it, and add a matching tls_config to the scrape job so internal self-signed deployments can opt out of verification.
    • correctness-reviewer, adversarial-reviewer, reliability-reviewer
  • smoke-tests/otel-collector/promql-remote-write.bats:10 -- The HTTPS path, the primary behavior of this change, has no automated coverage: the only https:// smoke service includes a path and exits at the path-rejection check before the scheme/TLS wiring takes effect, so a regression in scheme: https or insecure=false would pass CI.
    • Fix: Add a smoke service with a path-free https://host:port endpoint asserting scheme: https is logged; a full TLS receiver can stay a follow-up.
    • testing-reviewer, correctness-reviewer, security-reviewer
🔵 P3 nitpicks (3)
  • docker/otel-collector/entrypoint.sh:52 -- Validation rejects a path but not a query string, fragment, user:pass@ userinfo, or embedded whitespace, so a value like https://host:9363?token=x becomes the malformed target host:9363?token=x/write (author acknowledged this as won't-fix for operator-controlled config).
    • Fix: Reject values that are not a plain host:port / [ipv6]:port (query, fragment, userinfo, whitespace) for parity with the path check.
  • packages/api/src/opamp/controllers/opampController.ts:352 -- The :-http / :-true fallback comments claim older collector images stay on plain HTTP, but that only holds for a bare host:port; under version skew a URL-form value on an image without the scheme-stripping entrypoint resolves to http://https://…/write, a crash rather than a graceful downgrade.
    • Fix: Tighten the comment to note URL-form endpoints require the matching collector image, or validate the resolved endpoint shape.
  • docker/otel-collector/config.standalone.promql.yaml:16 -- The prometheusremotewrite exporter block is duplicated verbatim between this YAML and opampController.ts, kept in sync only by hand-maintained cross-reference comments.
    • Fix: Add a test comparing the YAML exporter block against buildOtelCollectorConfig().exporters.prometheusremotewrite so drift fails loudly.

Reviewers (11): correctness, security, adversarial, reliability, testing, maintainability, project-standards, kieran-typescript, previous-comments, agent-native, learnings-researcher.

Testing gaps:

  • entrypoint.sh scheme-parsing branches for an unsupported scheme and an empty host are uncovered; only the http:// and path-error branches are exercised.
  • No test asserts config.standalone.promql.yaml and opampController.ts emit equivalent exporter/TLS values from the same env vars.
  • HTTPS success and certificate-verification behavior rests on unit-level config-string assertions plus manual dev-stack verification; a TLS receiver fixture was scoped out by the author.

Pre-existing (not counted): the clickhouse scrape job is emitted without gating on CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT being set, carried over from the prior combined otelcol job.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 436 passed • 1 skipped • 1431s

Status Count
✅ Passed 436
❌ Failed 0
⚠️ Flaky 1
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT now accepts an http:// or
https:// URL as well as a bare host:port. entrypoint.sh splits off the
scheme and passes it to the prometheusremotewrite exporter through
HYPERDX_PROMETHEUS_SCHEME and HYPERDX_PROMETHEUS_TLS_INSECURE. Both the
standalone and OpAMP-managed configs read these, with defaults that
keep plain HTTP for older collector images.

A bare host:port behaves as before. A path or an unsupported scheme
now fails at startup with a clear error.
@karl-power
karl-power force-pushed the allow-https-hosts-for-external-prom-ql-endpoints branch from d5fe3ef to c22f6dc Compare October 12, 2026 09:31
@karl-power karl-power changed the title fix: support https for the PromQL remote-write endpoint fix: support https for the PromQL remote-write and scrape endpoint Oct 12, 2026
@karl-power
karl-power marked this pull request as ready for review October 12, 2026 09:40
@karl-power
karl-power requested a review from wrn14897 October 12, 2026 09:40
@github-actions github-actions Bot added the review/tier-4 Critical — deep review + domain expert sign-off label Oct 12, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🔴 Tier 4 — Critical

Touches authentication, tenancy data models, the public API or shipped database config — or substantially changes the query rendering engine, background tasks, the OTel pipeline, image build, or release CI.

Why this tier:

  • Background tasks or delivery pipeline substantially modified — 46 lines (bar: 30):
    • docker/otel-collector/config.standalone.promql.yaml
    • docker/otel-collector/entrypoint.sh

Review process: Deep review from a domain expert. Synchronous walkthrough may be required.
SLA: Schedule synchronous review within 2 business days.

Stats
  • Production files changed: 8
  • Production lines changed: 203 (+ 46 in test files, excluded from tier calculation)
  • Critical-path lines changed: 46
  • Branch: allow-https-hosts-for-external-prom-ql-endpoints
  • Author: karl-power

To override this classification, remove the review/tier-4 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

@pulpdrew pulpdrew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@kodiakhq
kodiakhq Bot merged commit ca8e3fd into main Oct 19, 2026
29 checks passed
@kodiakhq
kodiakhq Bot deleted the allow-https-hosts-for-external-prom-ql-endpoints branch October 19, 2026 12:06

This branch was successfully deployed

2 active deployments
Preview – hyperdx-oss — 391a09da Deployed Oct 19, 2026 by vercel[bot]
Preview – hyperdx-storybook — 391a09da Deployed Oct 19, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

automerge review/tier-4 Critical — deep review + domain expert sign-off

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants