Repository navigation
fix: support https for the PromQL remote-write and scrape endpoint - #3330
Conversation
🦋 Changeset detectedLatest commit: 391a09d The changes in this PR will be included in the next version bump. This PR includes changesets to release 3 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Duplicate code analysisNo new duplicate code introduced by this PR. jscpd compared this branch against |
| case "$CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT" in | ||
| https://*) | ||
| HYPERDX_PROMETHEUS_SCHEME=https | ||
| HYPERDX_PROMETHEUS_TLS_INSECURE=false |
There was a problem hiding this comment.
🔵 minor — HYPERDX_PROMETHEUS_TLS_INSECURE probably has no effect on the HTTP exporter
Consider dropping HYPERDX_PROMETHEUS_TLS_INSECURE, the ${env:...} placeholder for insecure in config.standalone.promql.yaml and opampController.ts, and the insecure: boolean → string type change, and let the URL scheme decide. For an HTTP client, configtls LoadTLSConfig returns a nil TLS config when insecure: true and no CA is set. Go's transport then still does verified TLS for an https:// URL using the system CAs, so the scheme alone already controls TLS. Verify this against the pinned collector version before removing.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
| HYPERDX_PROMETHEUS_SCHEME=http | ||
| HYPERDX_PROMETHEUS_TLS_INSECURE=true | ||
| case "$CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT" in | ||
| https://*) |
There was a problem hiding this comment.
🔵 minor — With https:// there is no way to trust a private CA or skip verification
The exporter only verifies against the system CAs, so a ClickHouse endpoint using a self-signed or internal-CA certificate (common for ClickHouse with TLS) still cannot receive metrics. The collector already reads CLICKHOUSE_TLS_CA_FILE and CLICKHOUSE_TLS_INSECURE_SKIP_VERIFY for the migrate tool (packages/otel-collector/cmd/migrate/main.go:71-75). Map those into tls.ca_file / tls.insecure_skip_verify in both exporter configs, or document that only publicly trusted certificates work.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
PR Review
1 finding(s): 🔴 0 critical · 🟠 0 major · 🔵 1 minor 1 posted as inline comment(s) on the changed lines. Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides. |
[Medium impact] The PR appears safe to merge; no new actionable defects were found. SummaryAdds HTTPS support for PromQL remote writes and OpAMP-managed ClickHouse scraping.
|
Deep Review✅ No critical issues found. No P0/P1 ship-blockers: the change is backward-compatible for bare 🟡 P2 -- recommended
🔵 P3 nitpicks (3)
Reviewers (11): correctness, security, adversarial, reliability, testing, maintainability, project-standards, kieran-typescript, previous-comments, agent-native, learnings-researcher. Testing gaps:
Pre-existing (not counted): the No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond |
E2E Test Results✅ All tests passed • 436 passed • 1 skipped • 1431s
Tests ran across 4 shards in parallel. |
CLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT now accepts an http:// or https:// URL as well as a bare host:port. entrypoint.sh splits off the scheme and passes it to the prometheusremotewrite exporter through HYPERDX_PROMETHEUS_SCHEME and HYPERDX_PROMETHEUS_TLS_INSECURE. Both the standalone and OpAMP-managed configs read these, with defaults that keep plain HTTP for older collector images. A bare host:port behaves as before. A path or an unsupported scheme now fails at startup with a clear error.
d5fe3ef to
c22f6dc
Compare
🔴 Tier 4 — CriticalTouches authentication, tenancy data models, the public API or shipped database config — or substantially changes the query rendering engine, background tasks, the OTel pipeline, image build, or release CI. Why this tier:
Review process: Deep review from a domain expert. Synchronous walkthrough may be required. Stats
|
Why
The PromQL remote-write exporter was hardcoded to
http://withinsecure: true, so an HTTPS-only endpoint couldn’t receive metrics. The exporter now takes its scheme fromCLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT, in both the standalone config and the OpAMP-managed config.What changed
docker/otel-collector/entrypoint.shacceptshost:port,http://host:portorhttps://host:port. It strips the scheme, puts the barehost:portback intoCLICKHOUSE_PROMETHEUS_METRICS_ENDPOINT(the value is also used as a scrape target, which can't take a scheme), and exportsHYPERDX_PROMETHEUS_SCHEMEandHYPERDX_PROMETHEUS_TLS_INSECURE.https://turns on TLS with certificate verification against the system CAs.http://or no scheme keeps today's plain HTTP.config.standalone.promql.yamlandopampController.tsbuild the exporter endpoint and TLS setting from those env vars. The fallbacks (:-http,:-true) keep collector images that predate this change on plain HTTP./write), an unsupported scheme or no host now fails at startup with a clear error, before migrations run.clickhousejob withscheme: ${env:HYPERDX_PROMETHEUS_SCHEME:-http}. The collector's own0.0.0.0:8888target stays in theotelcoljob, since it is always plain HTTP. TLS verification stays on. Scraped ClickHouse metrics now carryjob="clickhouse"(andservice.name="clickhouse") instead ofotelcol. Nothing in the repo filters on the old value.http://URL and for the path error.How to test
1. Startup script parsing (no Docker)
From the repo root, define a helper that runs just the endpoint-parsing block of
entrypoint.sh:Then run each case:
Expected:
ch-server:9363→http, insecuretrue, endpoint unchangedhttp://ch-server:9363/→http,true, endpointch-server:9363https://ch.example.com:9363→https,false, endpointch.example.com:9363ftp://x:1,https://x:9363/write,https://→ each prints an error2. Smoke tests (Docker +
brew install bats-core)3. Dev stack, OpAMP path
Start the dev stack with
yarn dev. Find the container names withdocker ps | grep -E 'otel-collector|ch-server'.In
docker-compose.dev.yml, setCLICKHOUSE_PROMETHEUS_METRICS_ENDPOINTon theotel-collectorservice (nototel-collector-json) to each value below. Stop and start the stack after each change, then wait ~2 minutes.Remote-write and scrape logs:
ClickHouse data (run twice a minute apart;
latestshould advance):Expected:
ch-server:9363(default):scheme: http,http://ch-server:9363/write,remote write request successful, noScrape failed. Aclickhouserow with a recentlatest, and a rising time series count.http://ch-server:9363: same as the default.https://ch-server:9363:scheme: https,https://ch-server:9363/writeandHTTP response to HTTPS clientfor remote-write. The scrape also fails withScrape failedforscrape_pool: clickhouse, targethttps://ch-server:9363/metrics. Both errors are the expected result: local ClickHouse only serves plain HTTP, so they show the collector now uses HTTPS for both. Theotelcoljob still scrapes fine.Revert
docker-compose.dev.ymlafterwards.Linear
Closes HDX-4633