Repository navigation
fix(app): stop event deltas selection failing on distributed tables - #3235
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
🦋 Changeset detectedLatest commit: fbfe4f7 The changes in this PR will be included in the next version bump. This PR includes changesets to release 3 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🔵 Tier 2 — Low RiskSmall, isolated change with no API route or data model modifications. Why this tier:
Review process: AI review + quick human skim (target: 5–15 min). Reviewer validates AI assessment and checks for domain-specific concerns. Stats
|
| useTableMetadata(tcFromChartConfig(config)); | ||
| const isPointerTable = tableMetadata?.isPointerTable === true; | ||
| const canUsePartIdsHint = tableMetadata != null && !isPointerTable; | ||
| const inOperator = isPointerTable ? 'GLOBAL IN' : 'IN'; |
There was a problem hiding this comment.
🔵 minor — When the metadata lookup fails, the aggregate path falls back to plain IN, which still fails on Distributed tables
If useTableMetadata errors, isPointerTable is false, so an aggregate valueExpr on a Distributed source still gets Timestamp IN (SELECT … FROM AggregatedTimestamps) and hits the same distributed_product_mode = 'deny' error. Only the PartIds part of the fallback is safe. On a local MergeTree table, GLOBAL IN / GLOBAL NOT IN returns the same result as IN / NOT IN. So emit the GLOBAL form unconditionally and delete the inOperator/notInOperator branch. The query is then correct whatever the metadata returns.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
| * only identify rows within one shard, so the hint is meaningless there. | ||
| */ | ||
| const { data: tableMetadata, isLoading: isTableMetadataLoading } = | ||
| useTableMetadata(tcFromChartConfig(config)); |
There was a problem hiding this comment.
🔵 minor — A failing metadata lookup holds selection queries behind React Query's default retries
useTableMetadata inherits the default retry: 3 (no override in pages/_app.tsx or hooks/useMetadata.tsx:310). While it retries, isLoading stays true, so isSelectionQueryEnabled is false and the chart shows a spinner for several seconds before the no-hint fallback runs. Pass { retry: false } or a single retry as the second argument to useTableMetadata, so the fallback runs promptly.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
PR Review✅ No issues found. |
Selecting an area in event deltas added a PartIds subquery with indexHint((_part, _part_offset) IN PartIds). On a Distributed table with two or more shards, ClickHouse rejects that as a double-distributed IN (distributed_product_mode = 'deny'). Part offsets are also only meaningful within one shard. Skip the PartIds hint on Distributed/Merge tables. It is only an index hint, so results are unchanged. Use GLOBAL IN / GLOBAL NOT IN for the AggregatedTimestamps filter on those tables, since it is a real filter and hits the same error.
0aab205 to
6645b20
Compare
E2E Test Results✅ All tests passed • 381 passed • 1 skipped • 1299s
Tests ran across 4 shards in parallel. |
| const canUsePartIdsHint = tableMetadata != null && !isPointerTable; | ||
| const inOperator = isPointerTable ? 'GLOBAL IN' : 'IN'; | ||
| const notInOperator = isPointerTable ? 'GLOBAL NOT IN' : 'NOT IN'; | ||
| const isSelectionQueryEnabled = hasSelection && !isTableMetadataLoading; |
There was a problem hiding this comment.
🟠 major — A failing metadata lookup holds selection queries back for about 7s while react-query retries
useTableMetadata runs with the app's default QueryClient (packages/app/pages/_app.tsx:53), which has no retry override. React-query v5 therefore retries a failed lookup 3 times with backoff (1s, 2s, 4s). isLoading stays true through all of those retries, so the outlier and inlier queries, and the chart's loading state, wait roughly 7s before falling back. This is the case the PR calls a harmless fallback. Pass { retry: false } or retry: 1 to useTableMetadata(tcFromChartConfig(config), …), or gate on isFetched/isError rather than waiting out the retries.
Nothing blocks merge automatically, but a maintainer will expect this fixed if it is a real defect in code this PR changes. If it is about surrounding code, reply and say so instead of patching. Do not widen the PR. How to respond
| */ | ||
| const { data: tableMetadata, isLoading: isTableMetadataLoading } = | ||
| useTableMetadata(tcFromChartConfig(config)); | ||
| const isPointerTable = tableMetadata?.isPointerTable === true; |
There was a problem hiding this comment.
🔵 minor — The metadata-failure fallback still uses plain IN, so aggregate selections on Distributed tables fail the same way
When the metadata query fails, tableMetadata is undefined. That makes isPointerTable false, so buildAggregateTimestampCondition emits Timestamp IN (SELECT … FROM AggregatedTimestamps). On a Distributed source this is the double-distributed subquery that distributed_product_mode='deny' rejects, so an aggregate valueExpr hits the original error. Dropping the PartIds hint in this case was correct, but the IN choice was not made safe to match. When metadata is unknown, use GLOBAL IN/GLOBAL NOT IN: on local tables GLOBAL returns the same result, it only sends the subquery result along. Add a test for this case with valueExpr: 'count()'.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
|
Deep Review✅ No critical issues found. The most severe hypothesis — that a failed or disabled 🟡 P2 -- recommended
🔵 P3 nitpicks (2)
Reviewers (5): correctness, testing, maintainability, performance, agent-native. Testing gaps: No end-to-end coverage that the emitted No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond |
If the table metadata lookup failed, aggregate selections used plain IN against AggregatedTimestamps, which ClickHouse rejects on Distributed tables. Only a table confirmed local now gets plain IN and the PartIds hint; anything else uses GLOBAL IN, which also works on local tables. With the fallback correct, the lookup no longer retries, so a failure doesn't hold the selection queries back through ~7s of backoff.
Duplicate code analysisNo new duplicate code introduced by this PR. jscpd compared this branch against |
Summary
Drawing a box on the event deltas heatmap fails on sources backed by a Distributed table with 2+ shards:
The selection queries add a
PartIdssubquery plusindexHint((_part, _part_offset) IN PartIds)to read fewer granules. On a Distributed table this breaks in two ways:_part/_part_offsetare per-shard, so they point at unrelated rows on other shards.The aggregate-value path (
Timestamp IN (SELECT … FROM AggregatedTimestamps)) fails the same way.Fix: when table metadata marks the table as Distributed or Merge (
isPointerTable):PartIdshint: dropped. It's only an index hint, so results are unchanged.AggregatedTimestampsfilter: usesGLOBAL IN/GLOBAL NOT IN.Normal tables are unchanged. Selection queries wait for metadata before running. If the lookup fails, queries are built as if the table is Distributed (
GLOBAL IN, no hint). That works on any table, just without the speed-up, so the lookup doesn't retry.Trade-off: selections on Distributed tables lose the speed-up.
distributed_product_mode = 'local'could keep it, but read-only ClickHouse users can't change settings.Local repro:
config.dwith shards127.0.0.1and127.0.0.2.otel_traces_distand draw a box in event deltas.How to test on Vercel preview
Preview routes: /search
Steps:
References