Skip to content

perf: compress the seed schemas with ZSTD(3) instead of ZSTD(1) - #3514

Merged
kodiakhq[bot] merged 2 commits into
ClickHouse:mainfrom
alexey-milovidov:seed-schemas-zstd3
Nov 2, 2026
Merged

kodiakhq[bot] merged 2 commits into
ClickHouse:mainfrom
alexey-milovidov:seed-schemas-zstd3

Conversation

@alexey-milovidov

Copy link
Copy Markdown
Member

Summary

The seed schemas in docker/otel-collector/schema/seed/ (and their copy in docker/clickhouse/local/init-db-e2e.sh) pin ZSTD(1) on almost every column. ZSTD(3) gives a noticeably better compression ratio for this kind of data at a small insert-time CPU cost, and decompression speed (so query performance) stays the same. It is also the default codec of ClickHouse since 26.9 (ClickHouse/ClickHouse@9c56196a1a4a).

The codec is pinned explicitly rather than removed, because the bundled ClickHouse is 26.8 and the _compat schemas target servers older than 26.2: on these versions the default is LZ4, so removing the codec would make compression worse.

  • CODEC(ZSTD(1)) → CODEC(ZSTD(3)), CODEC(T64, ZSTD(1)) → CODEC(T64, ZSTD(3)).
  • CODEC(Delta(8), ZSTD(1)) → CODEC(Delta, ZSTD(3)): the byte-size argument of Delta is inferred from the column type, so it is unneeded.

Only newly created tables are affected (CREATE TABLE IF NOT EXISTS); existing tables keep their codecs. Applied migrations in packages/api/migrations/ch/ and test fixtures are not changed.

How to reproduce

Starting point: any ClickHouse server, or clickhouse local.

On this branch:

  1. All changed seed files (00002–00005, including _compat) were loaded into ClickHouse with ${DATABASE} and ${*_TTL} substituted (toIntervalDay(30)); every table is created successfully.

References

The default codec in ClickHouse is ZSTD(3) since 26.9. Pin it explicitly,
so that servers older than 26.9 (default LZ4) get it too. Drop the
unneeded byte-size argument of Delta, which is inferred from the type.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@alexey-milovidov
alexey-milovidov requested a review from a team as a code owner November 1, 2026 23:44
@changeset-bot

changeset-bot Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 4fb555a

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages
Name Type
@hyperdx/otel-collector Patch
@hyperdx/api Patch
@hyperdx/app Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercel Bot commented Nov 1, 2026

Copy link
Copy Markdown

@alexey-milovidov is attempting to deploy a commit to the HyperDX Team on Vercel.

A member of the Team first needs to authorize it.

@github-actions github-actions Bot added external needs-vouch Author needs a maintainer to vouch for them labels Nov 1, 2026
@github-actions

github-actions Bot commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

Hi @alexey-milovidov, thanks for the pull request!

Before we review code from a first-time contributor, a maintainer vouches for them, and you're not on our list yet. There's nothing else to file and this PR stays open — a maintainer will look at it, usually within a day or two, and vouch for you right here if it's a genuine attempt at the problem.

To make that quick, check the PR description has:

  • The issue it addresses (ideally one labelled Accepting PR) and why the change is needed
  • Steps to reproduce the problem and verify the fix, starting from https://play.hyperdx.io or yarn app:dev:local connected to the demo server
  • Before/after screenshots if anything in the UI changed

More detail in our contributing guide.

@github-actions

github-actions Bot commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

PR Review

✅ No issues found.

@github-actions

github-actions Bot commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

Deep Review

✅ No critical issues found.

This is a mechanical, low-risk schema change: across the six ClickHouse seed files and the init-db-e2e.sh copy, per-column codecs move from CODEC(ZSTD(1)) → CODEC(ZSTD(3)) (and T64, ZSTD(1) → T64, ZSTD(3)), and the explicit byte-size argument is dropped from Delta codecs (Delta(8)/Delta(1) → Delta). All seed tables use CREATE TABLE IF NOT EXISTS, so only newly created tables are affected; existing tables keep their codecs and no data is rewritten.

Key correctness checks pass:

  • Dropping the Delta byte-size argument is semantically equivalent — ClickHouse infers the delta size from the column type, and every converted site matches (Delta(8) on 8-byte DateTime64(9)/UInt64, Delta(1) on 1-byte Bool).
  • ZSTD(3), T64, and argument-less Delta are all valid on the _compat schemas targeting ClickHouse < 26.2, so the compatibility path is unaffected.
  • No code or test in packages/ compares the seed DDL against the applied migrations, so the intentional seed-vs-migration codec divergence introduces no drift-check failure.
🔵 P3 nitpicks (2)
  • docker/clickhouse/local/init-db-e2e.sh:356 — the seed schemas (ZSTD(3)) now diverge from the applied migrations in packages/api/migrations/ch/ and the integration-test fixtures, which remain on ZSTD(1); this is deliberate and harmless but leaves the codec string defined in several places that can drift.
    • Fix: leave a short comment near the codecs in one canonical seed file noting that migrations and fixtures intentionally stay on ZSTD(1), so a future contributor does not "fix" the mismatch.
  • docker/clickhouse/local/init-db-e2e.sh:356 — init-db-e2e.sh is a hand-maintained duplicate of the docker/otel-collector/schema/seed/ schemas, so codec edits must be applied in two places and can silently drift (e.g. IsMonotonic already carries a Delta codec here but plain ZSTD in 00003_otel_metrics.sql — pre-existing).
    • Fix: track consolidating the e2e init script to source the seed schemas rather than copying them, as a follow-up.

Reviewers (6): correctness, testing, maintainability, project-standards, data-migrations, performance.

Testing gaps: No automated check confirms the seed SQL (00002–00005, including _compat) parses and creates cleanly on the bundled ClickHouse (26.8) and the < 26.2 compat path; validation here relies on the author's manual load. Consider a lightweight CI smoke-load of the seed files.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

@knudtty knudtty left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Alexey!

@kodiakhq
kodiakhq Bot merged commit 8384fe6 into ClickHouse:main Nov 2, 2026
25 of 27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

automerge external needs-vouch Author needs a maintainer to vouch for them

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants