Skip to content

[HDX-5860] Disable multipath TCP in the OTel collector - #3505

Merged
kodiakhq[bot] merged 2 commits into
mainfrom
warren/HDX-5860-disable-multipath-tcp
Oct 31, 2026
Merged

kodiakhq[bot] merged 2 commits into
mainfrom
warren/HDX-5860-disable-multipath-tcp

Conversation

@wrn14897

Copy link
Copy Markdown
Contributor

Summary

Mitigates CVE-2026-98432 for the OTel collector by disabling Multipath TCP in the Go runtime.

The CVE is a Linux kernel use-after-free in IPv4 TCP connection setup. The published remote-root exploit targets servers with an MPTCP listener. Since Go 1.24, net.Listen asks for MPTCP by default, so on a host with MPTCP enabled, every collector listener (4317, 4318, 13133, 8888) is one. The Go Security Team's runtime mitigation is GODEBUG=multipathtcp=0.

docker/otel-collector/entrypoint.sh now adds multipathtcp=0 to GODEBUG before anything else runs. Every Go process the image starts goes through this script, so this one change covers:

  • the collector in standalone mode (exec /otelcontribcol)
  • the OpAMP supervisor and the collector it spawns (the supervisor's commander leaves cmd.Env nil when no agent.env is configured, so the child inherits the environment)
  • the migrate tool
  • the all-in-one images, which run the same script as /otel-entrypoint.sh

The value is appended rather than set with ENV in the Dockerfile, so a deployment that passes its own GODEBUG (for example from a Helm chart) keeps its other settings and still gets the mitigation. An explicit multipathtcp= value from the operator is left alone.

Not changed: docker/hostmetrics (stock contrib 0.90.0 built with Go 1.21, where MPTCP is off by default, and not referenced by any compose file or workflow).

This only reduces exposure. The Go team calls per-program MPTCP disabling fragile, and the real fix is a patched host kernel.

How to reproduce

Starting point: repo root, Docker running.

On main:

  1. docker build -f docker/otel-collector/Dockerfile --target dev -t otel-mptcp .
  2. Start ClickHouse and the collector on a shared network (standalone mode, HYPERDX_OTEL_EXPORTER_CREATE_LEGACY_SCHEMA=true, CLICKHOUSE_ENDPOINT=tcp://<ch>:9000).
  3. docker exec <collector> sh -c 'tr "\0" "\n" < /proc/1/environ | grep GODEBUG' prints nothing.

On this branch:

  1. Same steps. The command prints GODEBUG=multipathtcp=0.
  2. Passing -e GODEBUG=http2client=0 gives GODEBUG=http2client=0,multipathtcp=0.
  3. In supervisor mode (OPAMP_SERVER_URL set), PID 1 opampsupervisor carries GODEBUG=multipathtcp=0.

I ran all three cases locally. I could not confirm the change at the socket level: Docker Desktop's VM kernel has no MPTCP support, so listeners are plain TCP either way. It's worth checking ss -ltnM on a real node after rollout.

Added smoke-tests/otel-collector/godebug.bats, which asserts the collector's PID 1 environment. I have not run it locally.

make ci-lint passes. make ci-unit passes apart from timeZone.test.ts (getTimeZoneShortName), which also fails on main on my machine (ICU/locale data) and is unrelated.

References

Go 1.24+ opens MPTCP listeners by default, exposing the collector to
CVE-2026-98432 on hosts with MPTCP enabled. The entrypoint now appends
multipathtcp=0 to GODEBUG, covering the collector, the OpAMP supervisor
and its child, and the migrate tool, in both the collector and
all-in-one images.
@wrn14897 wrn14897 added the ai-generated AI-generated content; review carefully before merging. label Oct 30, 2026
@vercel

vercel Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

2 Skipped Deployments
Project Deployment Actions Updated
hyperdx-oss Ignored Ignored Preview Oct 31, 2026 2:36pm UTC
hyperdx-storybook Ignored Ignored Preview Oct 31, 2026 2:36pm UTC

Request Review

@changeset-bot

changeset-bot Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 70823ef

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages
Name Type
@hyperdx/otel-collector Patch
@hyperdx/api Patch
@hyperdx/app Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions

Copy link
Copy Markdown
Contributor

Duplicate code analysis

No new duplicate code introduced by this PR.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 114 in total (1.40%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 461 passed • 1 skipped • 1325s

Status Count
✅ Passed 461
❌ Failed 0
⚠️ Flaky 1
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

@wrn14897
wrn14897 marked this pull request as ready for review October 30, 2026 22:54
@wrn14897
wrn14897 requested a review from a team as a code owner October 30, 2026 22:54
@wrn14897
wrn14897 requested a review from karl-power October 30, 2026 22:54
@github-actions github-actions Bot added the review/tier-2 Low risk — AI review + quick human skim label Oct 30, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🔵 Tier 2 — Low Risk

Small, isolated change with no API route or data model modifications.

Why this tier:

  • Standard feature/fix — introduces new logic or modifies core functionality

Additional context: touches background tasks or the delivery pipeline lightly (8 lines, under the 30-line bar for Tier 4)

Review process: AI review + quick human skim (target: 5–15 min). Reviewer validates AI assessment and checks for domain-specific concerns.
SLA: Resolve within 4 business hours.

Stats
  • Production files changed: 2
  • Production lines changed: 18
  • Branch: warren/HDX-5860-disable-multipath-tcp
  • Author: wrn14897

To override this classification, remove the review/tier-2 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

@test "GODEBUG: the collector process runs with multipathtcp=0" {
run docker compose exec -T otel-collector sh -c 'tr "\0" "\n" < /proc/1/environ'
[ "$status" -eq 0 ]
echo "$output" | grep -qx 'GODEBUG=multipathtcp=0'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Smoke test only covers the unset-GODEBUG standalone case, not the merge logic or the supervisor path

The only collector checked is the otel-collector service in smoke-tests/otel-collector/docker-compose.yaml, which has no GODEBUG set. So the test would still pass if the ${GODEBUG:+$GODEBUG,} append were broken (operator settings dropped or overwritten) or if the *multipathtcp=* case were inverted, and it never checks supervisor mode, where the protected listener belongs to a child process the supervisor spawns. Fix: add a compose service with GODEBUG=http2client=0 and assert GODEBUG=http2client=0,multipathtcp=0. Add one with GODEBUG=multipathtcp=1 and assert it is left as is. Ideally also run entrypoint.sh in supervisor mode and assert that GODEBUG reaches the spawned /otelcontribcol (read its /proc/<pid>/environ), not just PID 1.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

@github-actions

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 critical or 🟠 major in code this PR adds or changes: fix before asking for review.
  • 🔵 minor in code this PR changes: your call. Fix if small, otherwise reply.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    severity. Reply with one sentence; if it is critical, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

1 finding(s): 🔴 0 critical · 🟠 0 major · 🔵 1 minor

1 posted as inline comment(s) on the changed lines.


Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides.

@github-actions

Copy link
Copy Markdown
Contributor

Deep Review

✅ No critical issues found. The mitigation is effective and complete for the diff: the GODEBUG export runs before every exec/migrate invocation, so the standalone collector, the supervisor-spawned collector (supervisor_docker.yaml.tmpl sets no agent.env, so the child inherits), the migrate tool, and the all-in-one image's /otel-entrypoint.sh all receive multipathtcp=0. The case guard is correct across unset/empty/preset GODEBUG states.

🟡 P2 -- recommended

  • docker/otel-collector/entrypoint.sh:9 -- The append/merge branch (the operator-supplied-GODEBUG path the changeset explicitly promises to preserve) carries all the string-manipulation logic but has no test; the only smoke test runs in standalone mode where the compose service sets no GODEBUG, exercising solely the unset branch.
    • Fix: Add a bats case that runs the entrypoint with a preset GODEBUG (e.g. http2debug=1) and asserts the result preserves the original key and ends with ,multipathtcp=0.
    • testing, correctness, security
🔵 P3 nitpicks (3)
  • docker/otel-collector/entrypoint.sh:8 -- The *multipathtcp=* guard also matches a malformed empty value (GODEBUG=multipathtcp=), causing the entrypoint to skip the mitigation while Go treats the empty value as the 1.24+ default and keeps MPTCP enabled; narrow operator-misconfiguration path.
    • Fix: Match on an explicit value such as *multipathtcp=[01]* so a truncated value falls through to the injection branch.
    • security, correctness
  • smoke-tests/otel-collector/godebug.bats:9 -- The grep -qx 'GODEBUG=multipathtcp=0' full-line assertion will fail if a GODEBUG value is ever added to the compose service, even though the appended mitigation would still be present.
    • Fix: Assert the GODEBUG= line contains multipathtcp=0 rather than equals the whole string.
    • maintainability
  • docker/otel-collector/entrypoint.sh:8 -- The "explicit multipathtcp= value is left as is" no-op branch is an advertised contract in the changeset with no coverage.
    • Fix: Add a bats case asserting a preset GODEBUG=multipathtcp=1 is left unchanged.
    • testing

Reviewers (6): correctness, testing, reliability, maintainability, project-standards, security.

Testing gaps:

  • Supervisor mode (OPAMP_SERVER_URL set, PID 1 = opampsupervisor) env inheritance into the spawned child collector -- the load-bearing listener path -- is unverified by any smoke test.
  • The all-in-one image (/otel-entrypoint.sh via entry.local.base.sh) has no GODEBUG assertion.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

@MikeShi42
MikeShi42 self-requested a review October 31, 2026 14:35
@kodiakhq
kodiakhq Bot merged commit 7a9fe57 into main Oct 31, 2026
28 checks passed
@kodiakhq
kodiakhq Bot deleted the warren/HDX-5860-disable-multipath-tcp branch October 31, 2026 14:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-generated AI-generated content; review carefully before merging. automerge review/tier-2 Low risk — AI review + quick human skim

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants