Skip to content

fix(app): stop event deltas selection failing on distributed tables - #3235

Merged
kodiakhq[bot] merged 5 commits into
mainfrom
claude/delta-chart-distributed-tables
Sep 28, 2026
Merged

kodiakhq[bot] merged 5 commits into
mainfrom
claude/delta-chart-distributed-tables

Conversation

@karl-power

@karl-power karl-power commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Drawing a box on the event deltas heatmap fails on sources backed by a Distributed table with 2+ shards:

Double-distributed IN/JOIN subqueries is denied (distributed_product_mode = 'deny')

The selection queries add a PartIds subquery plus indexHint((_part, _part_offset) IN PartIds) to read fewer granules. On a Distributed table this breaks in two ways:

  • ClickHouse rejects it: it's an IN subquery over a Distributed table.
  • The hint is meaningless: _part / _part_offset are per-shard, so they point at unrelated rows on other shards.

The aggregate-value path (Timestamp IN (SELECT … FROM AggregatedTimestamps)) fails the same way.

Fix: when table metadata marks the table as Distributed or Merge (isPointerTable):

  • PartIds hint: dropped. It's only an index hint, so results are unchanged.
  • AggregatedTimestamps filter: uses GLOBAL IN / GLOBAL NOT IN.

Normal tables are unchanged. Selection queries wait for metadata before running. If the lookup fails, queries are built as if the table is Distributed (GLOBAL IN, no hint). That works on any table, just without the speed-up, so the lookup doesn't retry.

Trade-off: selections on Distributed tables lose the speed-up. distributed_product_mode = 'local' could keep it, but read-only ClickHouse users can't change settings.

Local repro:

  1. Add a 2-shard cluster in config.d with shards 127.0.0.1 and 127.0.0.2.
  2. Create the table:
    CREATE TABLE default.otel_traces_dist AS default.otel_traces
    ENGINE = Distributed('hdx_two_shards', 'default', 'otel_traces')
  3. Point a trace source at otel_traces_dist and draw a box in event deltas.

How to test on Vercel preview

Preview routes: /search

Steps:

  1. Select a trace source.
  2. Drag a box over part of the event deltas heatmap.
  3. Verify the "Selection" and "Background" legend appears and attribute charts render without an error.

References

  • Linear Issue: Closes HDX-5493

@vercel

vercel Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Sep 28, 2026 2:31pm UTC
hyperdx-storybook Ready Ready Preview Sep 28, 2026 2:31pm UTC

Request Review

@changeset-bot

changeset-bot Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: fbfe4f7

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages
Name Type
@hyperdx/app Patch
@hyperdx/api Patch
@hyperdx/otel-collector Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions github-actions Bot added the review/tier-2 Low risk — AI review + quick human skim label Sep 28, 2026
@github-actions

github-actions Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

🔵 Tier 2 — Low Risk

Small, isolated change with no API route or data model modifications.

Why this tier:

  • Standard feature/fix — introduces new logic or modifies core functionality

Review process: AI review + quick human skim (target: 5–15 min). Reviewer validates AI assessment and checks for domain-specific concerns.
SLA: Resolve within 4 business hours.

Stats
  • Production files changed: 1
  • Production lines changed: 113 (+ 130 in test files, excluded from tier calculation)
  • Branch: claude/delta-chart-distributed-tables
  • Author: karl-power

To override this classification, remove the review/tier-2 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

useTableMetadata(tcFromChartConfig(config));
const isPointerTable = tableMetadata?.isPointerTable === true;
const canUsePartIdsHint = tableMetadata != null && !isPointerTable;
const inOperator = isPointerTable ? 'GLOBAL IN' : 'IN';

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — When the metadata lookup fails, the aggregate path falls back to plain IN, which still fails on Distributed tables

If useTableMetadata errors, isPointerTable is false, so an aggregate valueExpr on a Distributed source still gets Timestamp IN (SELECT … FROM AggregatedTimestamps) and hits the same distributed_product_mode = 'deny' error. Only the PartIds part of the fallback is safe. On a local MergeTree table, GLOBAL IN / GLOBAL NOT IN returns the same result as IN / NOT IN. So emit the GLOBAL form unconditionally and delete the inOperator/notInOperator branch. The query is then correct whatever the metadata returns.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

* only identify rows within one shard, so the hint is meaningless there.
*/
const { data: tableMetadata, isLoading: isTableMetadataLoading } =
useTableMetadata(tcFromChartConfig(config));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — A failing metadata lookup holds selection queries behind React Query's default retries

useTableMetadata inherits the default retry: 3 (no override in pages/_app.tsx or hooks/useMetadata.tsx:310). While it retries, isLoading stays true, so isSelectionQueryEnabled is false and the chart shows a spinner for several seconds before the no-hint fallback runs. Pass { retry: false } or a single retry as the second argument to useTableMetadata, so the fallback runs promptly.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

@github-actions

github-actions Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

✅ No issues found.

Selecting an area in event deltas added a PartIds subquery with
indexHint((_part, _part_offset) IN PartIds). On a Distributed table with
two or more shards, ClickHouse rejects that as a double-distributed IN
(distributed_product_mode = 'deny'). Part offsets are also only
meaningful within one shard.

Skip the PartIds hint on Distributed/Merge tables. It is only an index
hint, so results are unchanged. Use GLOBAL IN / GLOBAL NOT IN for the
AggregatedTimestamps filter on those tables, since it is a real filter
and hits the same error.
@karl-power
karl-power force-pushed the claude/delta-chart-distributed-tables branch from 0aab205 to 6645b20 Compare September 28, 2026 13:24
@github-actions

github-actions Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 381 passed • 1 skipped • 1299s

Status Count
✅ Passed 381
❌ Failed 0
⚠️ Flaky 3
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

const canUsePartIdsHint = tableMetadata != null && !isPointerTable;
const inOperator = isPointerTable ? 'GLOBAL IN' : 'IN';
const notInOperator = isPointerTable ? 'GLOBAL NOT IN' : 'NOT IN';
const isSelectionQueryEnabled = hasSelection && !isTableMetadataLoading;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 major — A failing metadata lookup holds selection queries back for about 7s while react-query retries

useTableMetadata runs with the app's default QueryClient (packages/app/pages/_app.tsx:53), which has no retry override. React-query v5 therefore retries a failed lookup 3 times with backoff (1s, 2s, 4s). isLoading stays true through all of those retries, so the outlier and inlier queries, and the chart's loading state, wait roughly 7s before falling back. This is the case the PR calls a harmless fallback. Pass { retry: false } or retry: 1 to useTableMetadata(tcFromChartConfig(config), …), or gate on isFetched/isError rather than waiting out the retries.

Nothing blocks merge automatically, but a maintainer will expect this fixed if it is a real defect in code this PR changes. If it is about surrounding code, reply and say so instead of patching. Do not widen the PR. How to respond

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

addressed in 8511657

*/
const { data: tableMetadata, isLoading: isTableMetadataLoading } =
useTableMetadata(tcFromChartConfig(config));
const isPointerTable = tableMetadata?.isPointerTable === true;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — The metadata-failure fallback still uses plain IN, so aggregate selections on Distributed tables fail the same way

When the metadata query fails, tableMetadata is undefined. That makes isPointerTable false, so buildAggregateTimestampCondition emits Timestamp IN (SELECT … FROM AggregatedTimestamps). On a Distributed source this is the double-distributed subquery that distributed_product_mode='deny' rejects, so an aggregate valueExpr hits the original error. Dropping the PartIds hint in this case was correct, but the IN choice was not made safe to match. When metadata is unknown, use GLOBAL IN/GLOBAL NOT IN: on local tables GLOBAL returns the same result, it only sends the subquery result along. Add a test for this case with valueExpr: 'count()'.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

@greptile-apps

greptile-apps Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

[Medium risk] Fixes chart queries on distributed database tables.

The PR appears safe to merge based on the reviewed changes.

Summary

The PR adjusts event-deltas selection queries according to table metadata:

  • Distributed and Merge tables omit the shard-local PartIds hint and use GLOBAL IN for aggregate timestamp filtering.
  • Selection queries wait for metadata; if lookup fails, they use the hint-free, global-filter fallback.
  • Tests cover the generated queries for local, pointer, loading, and unavailable-metadata cases.
Diagram
%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Heatmap selection] --> B{Metadata loading?}
  B -- Yes --> C[Wait to query]
  B -- No --> D{Confirmed local table?}
  D -- Yes --> E[PartIds hint; ordinary IN]
  D -- No or lookup failed --> F[No PartIds hint; GLOBAL IN]
  E --> G[Selection and background queries]
  F --> G
Loading

Reviews (5) · Last reviewed commit: "Merge branch 'main' into claude/delta-ch..."

@github-actions

github-actions Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

✅ No critical issues found.

The most severe hypothesis — that a failed or disabled useTableMetadata lookup could either strand the selection queries in a permanent loading state or fall back to plain IN on a Distributed table (the prior greptile P1) — does not hold against the current code. @tanstack/react-query is v5, where isLoading = isPending && isFetching; an errored (retry: false) or disabled query reports isLoading: false, so isSelectionQueryEnabled becomes true and the fallback runs. isLocalTable = tableMetadata != null && !tableMetadata.isPointerTable routes the metadata-unavailable case to GLOBAL IN/GLOBAL NOT IN with the PartIds hint dropped, matching the local-table happy path only when metadata confirms a non-pointer engine. The non-aggregate pointer-table path correctly omits both the PartIds CTE and the indexHint filter under the same canUsePartIdsHint guard, leaving no dangling CTE reference.

🟡 P2 -- recommended

  • packages/app/src/components/__tests__/DBDeltaChart.test.tsx:191 -- New tests assert only the generated SQL condition and CTE strings through a mocked query hook; nothing verifies that ClickHouse actually accepts GLOBAL IN against a real multi-shard Distributed table, so the fix's core engine-contract claim is unverified end-to-end.
    • Fix: Add an integration test that runs the outlier/inlier selection against a Distributed-backed table, or record the unit-only coverage boundary so the gap is explicit.
🔵 P3 nitpicks (2)
  • packages/app/src/components/DBDeltaChart.tsx:121 -- canUsePartIdsHint is an exact alias of isLocalTable with no transformation, adding a second name that must stay mentally in sync with the first.
    • Fix: Reference isLocalTable directly at the partIdsCTE and indexHint guards, or drop the alias.
  • packages/app/src/components/DBDeltaChart.tsx:196 -- buildAggregateTimestampCondition reads inOperator/notInOperator, but inside partIdsCTE it only runs when isLocalTable is true, so its GLOBAL IN branch is unreachable there and can mislead a future reader debugging distributed behavior.
    • Fix: Leave the shared helper as-is for buildFilters, and optionally note the local-only invariant at the partIdsCTE call site.

Reviewers (5): correctness, testing, maintainability, performance, agent-native.

Testing gaps: No end-to-end coverage that the emitted GLOBAL IN query is accepted by a live Distributed/Merge table; the both-CTE-present state (aggregate + local table, where AggregatedTimestamps and PartIds coexist) is exercised only for its condition strings, not for the with array contents/ordering.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

If the table metadata lookup failed, aggregate selections used plain IN
against AggregatedTimestamps, which ClickHouse rejects on Distributed
tables. Only a table confirmed local now gets plain IN and the PartIds
hint; anything else uses GLOBAL IN, which also works on local tables.

With the fallback correct, the lookup no longer retries, so a failure
doesn't hold the selection queries back through ~7s of backoff.
@github-actions

Copy link
Copy Markdown
Contributor

Duplicate code analysis

No new duplicate code introduced by this PR.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 129 in total (1.84%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

@pulpdrew pulpdrew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, nice fix

@kodiakhq
kodiakhq Bot merged commit bd40e3c into main Sep 28, 2026
29 checks passed
@kodiakhq
kodiakhq Bot deleted the claude/delta-chart-distributed-tables branch September 28, 2026 14:37

This branch was successfully deployed

2 active deployments
Preview – hyperdx-oss — 4aaecbe1 Deployed Sep 28, 2026 by vercel[bot]
Preview – hyperdx-storybook — 4aaecbe1 Deployed Sep 28, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

automerge review/tier-2 Low risk — AI review + quick human skim

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants