Skip to content

feat: replicated schema migration - #3309

Open
SpencerTorres wants to merge 1 commit into
mainfrom
spencer/replicated-engine
Open

SpencerTorres wants to merge 1 commit into
mainfrom
spencer/replicated-engine

Conversation

@SpencerTorres

Copy link
Copy Markdown
Member

Summary

Adds opt-in support for ClickHouse's Replicated engines to the schema migration, so the database and tables can replicate across nodes.

  • HYPERDX_OTEL_EXPORTER_CLICKHOUSE_REPLICATED=true creates the database with the Replicated engine and seeds Replicated* tables. An existing non-Replicated database keeps plain engines, with a warning.
  • HYPERDX_OTEL_EXPORTER_CLICKHOUSE_FALLBACK_DATABASE keeps writing to an existing database when the configured one doesn't exist yet. The collector is pointed at whichever database the migration used.

Both are off by default, and nothing changes when unset. Tested with unit tests and against ClickHouse 25.7 and 26.2 with Keeper, including multi-node replication.

References

  • Linear Issue: HDX-4664

@changeset-bot

changeset-bot Bot commented Oct 7, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: f07caf7

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages
Name Type
@hyperdx/otel-collector Minor
@hyperdx/api Minor
@hyperdx/app Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercel Bot commented Oct 7, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

2 Skipped Deployments
Project Deployment Actions Updated
hyperdx-oss Ignored Ignored Oct 7, 2026 8:55am UTC
hyperdx-storybook Ignored Ignored Oct 7, 2026 8:55am UTC

Request Review

@github-actions github-actions Bot added the review/tier-4 Critical — deep review + domain expert sign-off label Oct 7, 2026
@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

🔴 Tier 4 — Critical

Touches authentication, tenancy data models, the public API or shipped database config — or substantially changes the query rendering engine, background tasks, the OTel pipeline, image build, or release CI.

Why this tier:

  • Background tasks or delivery pipeline substantially modified — 421 lines (bar: 30):
    • docker/otel-collector/entrypoint.sh
    • docker/otel-collector/schema/seed/00001_create_database.sql
    • docker/otel-collector/schema/seed/00002_otel_logs.sql
    • docker/otel-collector/schema/seed/00002_otel_logs_compat.sql
    • docker/otel-collector/schema/seed/00003_otel_metrics.sql
    • docker/otel-collector/schema/seed/00004_hyperdx_sessions.sql
    • docker/otel-collector/schema/seed/00005_otel_traces.sql
    • docker/otel-collector/schema/seed/00005_otel_traces_compat.sql
    • docker/otel-collector/schema/seed/00006_otel_logs_rollups.sql
    • docker/otel-collector/schema/seed/00007_otel_traces_rollups.sql
    • packages/otel-collector/cmd/migrate/main.go
    • packages/otel-collector/cmd/migrate/main_test.go

Review process: Deep review from a domain expert. Synchronous walkthrough may be required.
SLA: Schedule synchronous review within 2 business days.

Stats
  • Production files changed: 13
  • Production lines changed: 428
  • Critical-path lines changed: 421
  • Branch: spencer/replicated-engine
  • Author: SpencerTorres

To override this classification, remove the review/tier-4 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

Duplicate code analysis

No new duplicate code introduced by this PR.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 128 in total (1.79%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.


// A fallback database keeps the plain engines it was seeded with.
replicated := false
if database == cfg.Database {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Replicated mode is dropped without a warning when the fallback database is used

With HYPERDX_OTEL_EXPORTER_CLICKHOUSE_REPLICATED=true and a fallback that wins (the target is missing and the fallback has tables), planReplicated never runs. The fallback is seeded with plain engines and nothing says Replicated was not applied. Log a warning in that branch, for example: if cfg.Replicated && database != cfg.Database { log.Printf("WARNING: Replicated engines not applied; staying on fallback %s", database) }.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 critical or 🟠 major in code this PR adds or changes: fix before asking for review.
  • 🔵 minor in code this PR changes: your call. Fix if small, otherwise reply.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    severity. Reply with one sentence; if it is critical, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

3 finding(s): 🔴 0 critical · 🟠 0 major · 🔵 3 minor

1 posted as inline comment(s) on the changed lines. 2 listed below.

Findings outside the changed lines

2 minor
  • 🔵 packages/otel-collector/cmd/migrate/main.go:355 — The "not migrated" warning repeats on every startup, even for a target that is already in use → Without the replicated or fallback flags, dbExists is never queried and stays false. So resolveDatabase warns whenever default has ClickStack tables, even when the target database has been seeded and in use for a long time. One example is the smoke-test compose (smoke-tests/otel-collector/docker-compose.yaml): several collectors share one ClickHouse and one of them uses default. Always call getDatabaseEngine and only warn when !targetExists. Also log the hasClickStackTables error instead of dropping it when no fallback is set.
  • 🔵 packages/otel-collector/cmd/migrate/main.go:273 — The Replicated database Keeper path is hardcoded and cannot be changed later → /clickhouse/databases/<name> is fixed, so two deployments that share one Keeper ensemble (e.g. staging and prod, both using database otel) would join the same Replicated database and replicate DDL and data between them. The path also cannot change once a database is created. Expose the Keeper path through an env var (for example HYPERDX_OTEL_EXPORTER_CLICKHOUSE_REPLICATED_PATH), keeping the current value as the default.

Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides.

@greptile-apps

greptile-apps Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 4/5

[Critical risk] Database schema migration with replication and fallback logic.

Remove the prohibited test section banners before merging; no behavioral defect was established.

Fix All in Claude CodeFindings

  1. P2 Test banners break the guide ▶

Summary

Adds opt-in replicated database and table engines, plus a fallback database that is passed to the collector after seeding.

  • Existing non-replicated target databases keep plain table engines.
  • Seed statements and TTL updates use the selected database.
  • Remove the new test section banners to satisfy the repository’s comment rule.
Diagram
%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Read target and fallback settings] --> B{Target missing and fallback has tables?}
  B -->|Yes| C[Select fallback with plain engines]
  B -->|No| D[Select target]
  D --> E{Replication enabled and target missing or Replicated?}
  E -->|Yes| F[Use replicated engines]
  E -->|No| G[Use plain engines]
  C --> H[Seed selected database]
  F --> H
  G --> H
  H --> I{Fallback setting enabled?}
  I -->|Yes| J[Write selected database to handoff file]
  J --> K[Entrypoint exports database for collector]
  I -->|No| L[Keep configured collector database]
Loading

Reviews (1) · Last reviewed commit: "feat: replicated schema migration" · Reviewed by Greptile

Comment on lines +828 to +830
// ---------------------------------------------------------------------------
// replicated engines
// ---------------------------------------------------------------------------

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Test banners break the guide

The new section dividers violate the AGENTS.md directive: “No banner comments, no ASCII section dividers.” This rule also applies to tests. Remove this banner and the matching fallback banner at lines 974–976 before merging to satisfy the repository requirement.

Context Used: AGENTS.md (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Fix in Claude Code Fix in Conductor Fix in Cursor Fix in Codex

@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

Deep Review

Scope: PR #3309 feat: replicated schema migration — opt-in ClickHouse Replicated database/table engines plus an opt-in FALLBACK_DATABASE handshake in the Go migrate tool and collector entrypoint.sh. Diff base 0c1dd86. Both features are off by default.

✅ No critical issues found. No P0/P1: both features are opt-in, the fallback write path is fail-closed (log.Fatalf → set -e abort prevents the collector from starting against an unseeded database), and planReplicated refuses to replicate a pre-existing non-Replicated database. The items below are recommended hardening and test gaps.

🟡 P2 — recommended

  • docker/otel-collector/schema/seed/00008_otel_metrics_timeseries.sql:3 — metrics_ts is seeded unconditionally with ENGINE = TimeSeries and no ${ENGINE_PREFIX}, so with REPLICATED=true a non-Replicated experimental engine is created inside a Replicated database; TestSeedSchemaEngines explicitly skips this engine, and a rejected CREATE burns all retries and exits the seed non-zero, blocking collector startup.
    • Fix: Verify the TimeSeries table against a real Replicated database, and either exclude it from Replicated seeds or provide a replicated-compatible variant.
  • packages/otel-collector/cmd/migrate/main.go:559 — the Replicated DDL hardcodes the Keeper path Replicated('/clickhouse/databases/<db>', '{shard}', '{replica}') namespaced only by database name, so two independent clusters sharing one Keeper ensemble with the same database name cross-register as replicas, and recreating a database after DROP DATABASE can attach to stale Keeper metadata.
    • Fix: Incorporate a per-installation/cluster identifier (or {uuid}) into the znode path, and document the clean-znode requirement when reusing a database name.
    • adversarial, data-migrations
  • packages/otel-collector/cmd/migrate/main.go:908 — the composed main() decision (resolveDatabase + planReplicated + the if database == cfg.Database gate that forces replicated=false on the fallback path) has no test, though each helper is unit-tested in isolation; the REPLICATED=true + fallback-selected combination is the highest-value untested behavioral branch.
    • Fix: Extract the wiring into a pure function and table-test the flag/database-state combinations, including Replicated=true + fallback-selected ⇒ replicated=false.
    • testing, correctness, adversarial, maintainability
🔵 P3 nitpicks (6)
  • packages/otel-collector/cmd/migrate/main.go:908 — when the fallback database is selected, replicated is forced false and the fallback's own engine is never queried, so REPLICATED=true is silently dropped with no log stating replication was requested but skipped.
    • Fix: Query the resolved database's engine and log an explicit note when a requested Replicated seed is not applied because the fallback was chosen.
  • packages/otel-collector/cmd/migrate/main.go:888 — for any deployment with a non-default HYPERDX_OTEL_EXPORTER_CLICKHOUSE_DATABASE and no opt-in flags set, hasClickStackTables now issues a system.tables query against default on every startup and may emit a new WARNING, a behavior change versus the "nothing changes when unset" intent.
    • Fix: Gate the legacy-table probe behind the opt-in flags, or document the new always-on warning for custom-database users.
  • packages/otel-collector/cmd/migrate/main.go:897 — the new pre-migration probes (getDatabaseEngine, hasClickStackTables) log.Fatalf on any transient ClickHouse error and are not wrapped in the retry/backoff that guards the actual seed, so a momentary query failure after a successful ping hard-exits into a container restart loop.
    • Fix: Run the probes through the existing retry helper, or treat a probe error as non-fatal and fall back to non-replicated/no-fallback behavior.
  • packages/otel-collector/cmd/migrate/main.go:690 — a ResolvedDatabasePath write failure is fatal when the fallback diverged, so a persistently unwritable path (read-only /tmp, bad override, path-is-directory) re-seeds and dies every restart.
    • Fix: Write/validate the resolved-database file (or its parent dir) before the seed so a path problem fails fast without redoing migration work.
    • reliability, adversarial
  • docker/otel-collector/entrypoint.sh:40 — rm -f "$RESOLVED_DATABASE_PATH" runs under set -e; rm -f still returns non-zero when the path is a directory or on a read-only mount, aborting the entrypoint before migrate or the collector start.
    • Fix: Use rm -f "$RESOLVED_DATABASE_PATH" || true and/or validate the path points at a writable regular-file location.
  • packages/otel-collector/cmd/migrate/main.go:889 — the best-effort legacy check reuses the shared err variable and only acts on it when a fallback is configured, leaving a non-nil err assigned and silently carried until the next assignment overwrites it.
    • Fix: Use a local error variable (e.g. legacyHasTables, checkErr := ...) so the intentional swallow is explicit and no stale err is carried forward.

Reviewers (11): correctness, testing, maintainability, project-standards, data-migrations, reliability, adversarial, deployment-verification, previous-comments, learnings-researcher, agent-native.

Testing gaps:

  • No integration test creates a real Replicated database / ReplicatedMergeTree tables or exercises the {shard}/{replica} macro and Keeper-path lifecycle (incl. DROP + re-seed); coverage stops at macro-substitution string assertions.
  • The entrypoint.sh handshake (rm -f, -s guard, $(cat) newline stripping, env re-export) and the main() fatal branches (getDatabaseEngine/hasClickStackTables/WriteFile failures) have no automated coverage.
  • resolveDatabase when the legacy database holds only metrics/sessions tables (not otel_logs/otel_traces) is unexercised — hasClickStackTables would miss it and skip the fallback.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 398 passed • 1 skipped • 1523s

Status Count
✅ Passed 398
❌ Failed 0
⚠️ Flaky 1
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

@wrn14897 wrn14897 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you elaborate on why we need HYPERDX_OTEL_EXPORTER_CLICKHOUSE_FALLBACK_DATABASE?

func hasClickStackTables(ctx context.Context, db *sql.DB, database string) (bool, error) {
var n uint64
err := db.QueryRowContext(ctx,
"SELECT count() FROM system.tables WHERE database = ? AND name IN ('otel_logs', 'otel_traces')", database).Scan(&n)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we want to check only logs and traces here?

-- +goose Up
-- +goose NO TRANSACTION
CREATE DATABASE IF NOT EXISTS ${DATABASE};
CREATE DATABASE IF NOT EXISTS ${DATABASE}${DATABASE_ENGINE};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the possible value of DATABASE_ENGINE here?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

review/tier-4 Critical — deep review + domain expert sign-off

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants