Repository navigation
[HDX-5598] fix(common-utils): fall through to mapKeys scan when text index read fails - #3268
Conversation
…fails getMapKeys returned [] whenever its mergeTreeTextIndex query threw, so the rollup and bounded mapKeys scan paths never ran. ClickHouse refuses mergeTreeTextIndex with ACCESS_DENIED on any table with a row policy (ClickHouse/ClickHouse#119032), which left Map key autocomplete empty on those tables. Log and fall through to the next strategy instead. Fixes #3265 Refs HDX-5598
🦋 Changeset detectedLatest commit: fbd16b3 The changes in this PR will be included in the next version bump. This PR includes changesets to release 4 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Duplicate code analysis1 new clone introduced by this PR.
If one of these duplicates a helper that already exists, import it instead jscpd compared this branch against |
🔵 Tier 2 — Low RiskSmall, isolated change with no API route or data model modifications. Why this tier:
Additional context: touches the query rendering engine lightly (12 lines, under the 150-line bar for Tier 4) Review process: AI review + quick human skim (target: 5–15 min). Reviewer validates AI assessment and checks for domain-specific concerns. Stats
|
| @@ -1028,11 +1028,12 @@ export class Metadata { | |||
| return keys; | |||
| } | |||
| } catch (e) { | |||
There was a problem hiding this comment.
🔵 minor — A failed text-index read is retried for every new date range, not just once
cacheKey includes the date range whenever timestampValueExpression is set (lines 968-979), so the fall-through result only stops retries for that exact range. Every new search range, and the default range that moves forward each hour, sends the mergeTreeTextIndex query again, and it fails again with ACCESS_DENIED before the scan runs. That contradicts the 'one failed query before the result is cached' claim in the PR description. To fix it, cache the failure itself under a per-table key (e.g. ${connectionId}.${databaseName}.${tableName}.${column}.textIndexUnavailable) and skip both text-index branches while that key is set. Also extend the cache test to call again with a shifted dateRange and check that no new mergeTreeTextIndex( query is sent.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
PR Review✅ No issues found. |
|
Deep ReviewScope: The change is well-constructed. The two earlier automated P1s — an unbounded text-index read, and ✅ No critical (P0/P1) issues found. The items below are recommendations and nits. 🟡 P2 — recommended
🔵 P3 nitpicks (3)
Reviewers (11): correctness, reliability, security, performance, testing, maintainability, kieran-typescript, adversarial, previous-comments, project-standards, learnings. Testing gaps: The rollup-success-after-failure retry/caching path (non-access-denied failure where the rollup returns keys) is not covered; add a case asserting the failing index query is not re-issued on later calls through that path. No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond |
E2E Test Results✅ All tests passed • 392 passed • 1 skipped • 1629s
Tests ran across 4 shards in parallel. |
…d text index reads Rollup tables are separate tables that a row policy on the source table does not cover. Once mergeTreeTextIndex fails with ACCESS_DENIED we know a policy applies, so skip the rollup and read keys only from the policy-filtered source-table scan. Other text index failures still try the rollup first. Also cap the text index read at 15s (break on overflow, matching the other getMapKeys strategies) and forward the abort signal, so a slow index read plus the scan fallback can't stack up. Refs HDX-5598
| } catch (e) { | ||
| // e.g. ACCESS_DENIED under a row policy or BAD_ARGUMENTS on a | ||
| // Distributed table; the paths below still work. | ||
| rowPolicyDenied = isAccessDeniedError(e); |
There was a problem hiding this comment.
🔵 minor — Aborting during the text index read now falls through and caches an empty rollup result with no TTL
Before this PR the text index query didn't get abort_signal, so it couldn't be cancelled. Now it does, and both catch blocks treat a cancelled request as an ordinary failure. The call then moves on to the rollup query with a signal that is already aborted. The rollup getOrFetch catches that error and caches [] under its .rollup key (metadata.ts:1102-1156). That cache has no TTL, so later non-aborted calls skip a custom Map-indexing rollup for that range and always run the source scan. Every cancellation also logs a misleading "falling through" warning. describeSource.ts:240 passes signal today, so this path is reachable. Fix: in both catches, check for abort before setting rowPolicyDenied. If signal?.aborted, rethrow, and add a test for that case.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
timeout_overflow_mode 'break' would return a partial key list as a success and write it to the TTL-less metadata cache, and the pinned max_execution_time overrode the deployment's own setting. Planning-time failures like ACCESS_DENIED don't need it. Keep forwarding the abort signal. Refs HDX-5598
…th throw-on-timeout An aborted text index read used to fall through to the rollup with a dead signal; the rollup rejected at once, returned [], and getOrFetch cached that under the rollup key with no TTL, disabling the rollup for that column and range for the life of the process (reachable from the MCP describe_source timeout). Rethrow when the signal is aborted. Cap the index read at 15s with timeout_overflow_mode 'throw', matching the rollup and scan caps, so a slow read falls through promptly. 'throw' rather than 'break' so a timeout never caches a partial key list. Refs HDX-5598
Summary
Metadata.getMapKeysreturned[]whenever itsmergeTreeTextIndexquery threw, so the rollup table and boundedmapKeysscan paths never ran.Since ClickHouse/ClickHouse#119032 (backported to 26.6), ClickHouse refuses
mergeTreeTextIndexwithACCESS_DENIED(497) on any table with a row policy. On those tables search bar autocomplete showed no Map keys. The same short-circuit also hides keys on Distributed tables (BAD_ARGUMENTS, #2880).The fix: both text-index
catchblocks ingetMapKeysnow log and fall through to the next strategy, the same as when the index query succeeds with zero keys. This makesgetMapKeysconsistent withgetMapTextIndexKeyValuesandgetTextIndexKeyValues, which already skip a failed strategy. The scan reads the table itself, so it respects row policies. Its result is cached under the normal key, so the failing index read isn't re-issued on later calls.Row policies
Rollup tables (
metadataMaterializedViews) are separate tables, and a row policy on the source table doesn't cover them. When the text index read fails withACCESS_DENIED, we know a policy applies, sogetMapKeysskips the rollup and reads keys only from the source-table scan, which ClickHouse filters by the policy. Failures that aren't access denials (e.g.BAD_ARGUMENTSon Distributed tables) still try the rollup first. Detection uses the client error'stype/code(ACCESS_DENIED/497), with a message match as a fallback (isAccessDeniedErrorinclickhouse/index.ts).The shipped OTel rollups only index native columns, so on the default schema they never return Map keys. This only matters for custom rollups that aggregate Map columns. Some paths still read the rollup with no policy signal: no text index, no
system.data_skipping_indicesgrant, or ClickHouse < 26.3. This PR doesn't change them, so admins using custom Map-indexing rollups should mirror the source's row policies onto the rollup tables.Aborts and slow reads
If the caller aborts during the text index read,
getMapKeysrethrows instead of falling through. Otherwise the rollup would run with a dead signal, reject at once, and have its[]cached under the rollup key with no TTL. The index read is capped atmax_execution_time: 15withtimeout_overflow_mode: 'throw', matching the rollup and scan caps. A slow read falls through to rollup/scan promptly, and'throw'(not'break') means a timeout never caches a partial key list.Tests cover a key index failure, a kv index failure, the rollup being skipped after an ACCESS_DENIED (from either the error
causeor its message), the rollup still being tried for other failures, the 15s throw-on-timeout cap and abort signal, a timeout falling through to the rollup, an abort rethrowing without running or caching the fallback, and the fallback result being cached. They fail when the fix is reverted.Follow-ups tracked on the Linear ticket, not in this PR:
mergeTreeTextIndexstricter when a row policy applies ClickHouse#119032, Feature: Visualization of Secondary Index Content ClickHouse#99507)How to test on Vercel preview
N/A — non-UI change. To reproduce locally, follow the steps in #3265: create a row policy on
otel_logsfor a user that can readsystem.data_skipping_indices, then typeLogAttributes.in the search bar. Keys now appear.References