Skip to content

[HDX-5598] fix(common-utils): fall through to mapKeys scan when text index read fails - #3268

Merged
kodiakhq[bot] merged 5 commits into
mainfrom
warren/HDX-5598-map-keys-text-index-fallthrough
Oct 2, 2026
Merged

kodiakhq[bot] merged 5 commits into
mainfrom
warren/HDX-5598-map-keys-text-index-fallthrough

Conversation

@wrn14897

@wrn14897 wrn14897 commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Metadata.getMapKeys returned [] whenever its mergeTreeTextIndex query threw, so the rollup table and bounded mapKeys scan paths never ran.

Since ClickHouse/ClickHouse#119032 (backported to 26.6), ClickHouse refuses mergeTreeTextIndex with ACCESS_DENIED (497) on any table with a row policy. On those tables search bar autocomplete showed no Map keys. The same short-circuit also hides keys on Distributed tables (BAD_ARGUMENTS, #2880).

The fix: both text-index catch blocks in getMapKeys now log and fall through to the next strategy, the same as when the index query succeeds with zero keys. This makes getMapKeys consistent with getMapTextIndexKeyValues and getTextIndexKeyValues, which already skip a failed strategy. The scan reads the table itself, so it respects row policies. Its result is cached under the normal key, so the failing index read isn't re-issued on later calls.

Row policies

Rollup tables (metadataMaterializedViews) are separate tables, and a row policy on the source table doesn't cover them. When the text index read fails with ACCESS_DENIED, we know a policy applies, so getMapKeys skips the rollup and reads keys only from the source-table scan, which ClickHouse filters by the policy. Failures that aren't access denials (e.g. BAD_ARGUMENTS on Distributed tables) still try the rollup first. Detection uses the client error's type/code (ACCESS_DENIED/497), with a message match as a fallback (isAccessDeniedError in clickhouse/index.ts).

The shipped OTel rollups only index native columns, so on the default schema they never return Map keys. This only matters for custom rollups that aggregate Map columns. Some paths still read the rollup with no policy signal: no text index, no system.data_skipping_indices grant, or ClickHouse < 26.3. This PR doesn't change them, so admins using custom Map-indexing rollups should mirror the source's row policies onto the rollup tables.

Aborts and slow reads

If the caller aborts during the text index read, getMapKeys rethrows instead of falling through. Otherwise the rollup would run with a dead signal, reject at once, and have its [] cached under the rollup key with no TTL. The index read is capped at max_execution_time: 15 with timeout_overflow_mode: 'throw', matching the rollup and scan caps. A slow read falls through to rollup/scan promptly, and 'throw' (not 'break') means a timeout never caches a partial key list.

Tests cover a key index failure, a kv index failure, the rollup being skipped after an ACCESS_DENIED (from either the error cause or its message), the rollup still being tried for other failures, the 15s throw-on-timeout cap and abort signal, a timeout falling through to the rollup, an abort rethrowing without running or caching the fallback, and the fallback result being cached. They fail when the fix is reverted.

Follow-ups tracked on the Linear ticket, not in this PR:

How to test on Vercel preview

N/A — non-UI change. To reproduce locally, follow the steps in #3265: create a row policy on otel_logs for a user that can read system.data_skipping_indices, then type LogAttributes. in the search bar. Keys now appear.

References

…fails

getMapKeys returned [] whenever its mergeTreeTextIndex query threw, so the
rollup and bounded mapKeys scan paths never ran. ClickHouse refuses
mergeTreeTextIndex with ACCESS_DENIED on any table with a row policy
(ClickHouse/ClickHouse#119032), which left Map key autocomplete empty on
those tables. Log and fall through to the next strategy instead.

Fixes #3265
Refs HDX-5598
@wrn14897 wrn14897 added the ai-generated AI-generated content; review carefully before merging. label Oct 2, 2026
@changeset-bot

changeset-bot Bot commented Oct 2, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: fbd16b3

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages
Name Type
@hyperdx/common-utils Patch
@hyperdx/app Patch
@hyperdx/api Patch
@hyperdx/otel-collector Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercel Bot commented Oct 2, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hyperdx-oss Ready Ready Preview Oct 2, 2026 9:39am UTC
hyperdx-storybook Ready Ready Preview Oct 2, 2026 9:39am UTC

Request Review

@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Duplicate code analysis

1 new clone introduced by this PR.

  • packages/common-utils/src/core/metadata.ts:1022-1048 ↔ packages/common-utils/src/core/metadata.ts:1068-1091 — 27 lines

If one of these duplicates a helper that already exists, import it instead
of redefining it. If the duplication is deliberate, ignore this comment —
the check is advisory and never fails the build.


jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 127 in total (1.80%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

@wrn14897
wrn14897 marked this pull request as ready for review October 2, 2026 01:21
@github-actions github-actions Bot added the review/tier-2 Low risk — AI review + quick human skim label Oct 2, 2026
@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

🔵 Tier 2 — Low Risk

Small, isolated change with no API route or data model modifications.

Why this tier:

  • Standard feature/fix — introduces new logic or modifies core functionality

Additional context: touches the query rendering engine lightly (12 lines, under the 150-line bar for Tier 4)

Review process: AI review + quick human skim (target: 5–15 min). Reviewer validates AI assessment and checks for domain-specific concerns.
SLA: Resolve within 4 business hours.

Stats
  • Production files changed: 2
  • Production lines changed: 49 (+ 269 in test files, excluded from tier calculation)
  • Branch: warren/HDX-5598-map-keys-text-index-fallthrough
  • Author: wrn14897

To override this classification, remove the review/tier-2 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

@@ -1028,11 +1028,12 @@ export class Metadata {
return keys;
}
} catch (e) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — A failed text-index read is retried for every new date range, not just once

cacheKey includes the date range whenever timestampValueExpression is set (lines 968-979), so the fall-through result only stops retries for that exact range. Every new search range, and the default range that moves forward each hour, sends the mergeTreeTextIndex query again, and it fails again with ACCESS_DENIED before the scan runs. That contradicts the 'one failed query before the result is cached' claim in the PR description. To fix it, cache the failure itself under a per-table key (e.g. ${connectionId}.${databaseName}.${tableName}.${column}.textIndexUnavailable) and skip both text-index branches while that key is set. Also extend the cache test to call again with a shifted dateRange and check that no new mergeTreeTextIndex( query is sent.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

✅ No issues found.

@greptile-apps

greptile-apps Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 4/5

[Medium risk] Adds fallback logic when text index queries fail.

The PR does not appear safe to merge until auto-picked trace waterfalls either include roots beyond the probe margin or identify the resulting partial tree.

Summary

The PR makes Map-key discovery fall through after a failed text-index read, skips the key rollup after an access denial, and bounds index reads. Changes merged since the previous review also add time-bounded MCP trace-waterfall queries, reorganize the heatmap, extend PromQL reduction to categorical charts, and update release metadata.

  • Auto-picked traces can still be reported as complete when their root falls outside the new extent-probe margin.
Diagram
%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Auto-pick a span in the requested window] --> B[Probe trace ID within window ±6 hours]
  B --> C[Derive fetch bounds from spans found]
  C --> D[Fetch bounded span tree]
  B -. Root earlier than probe margin .-> E[Root omitted from derived bounds]
  E --> D
Loading

Reviews (5) · Last reviewed commit: "Merge branch 'main' into warren/HDX-5598..."

@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

Scope: packages/common-utils/src/core/metadata.ts (getMapKeys) and packages/common-utils/src/clickhouse/index.ts (isAccessDeniedError), plus their tests. Diff base: merge-base 5bdb68db. Mode: report-only (no edits/artifacts).

The change is well-constructed. The two earlier automated P1s — an unbounded text-index read, and timeout_overflow_mode: 'break' caching a partial key list — are resolved in the current HEAD: the index read is now capped at max_execution_time: 15 with timeout_overflow_mode: 'throw', verified at metadata.ts:998-1004. Abort rethrow, the !rowPolicyDenied rollup guard, and fallback caching are all covered by new tests.

✅ No critical (P0/P1) issues found. The items below are recommendations and nits.

🟡 P2 — recommended

  • packages/common-utils/src/core/metadata.ts:1101 — After a non-access-denied text-index failure (e.g. BAD_ARGUMENTS on a Distributed table), the rollup path still runs; rollup tables are not covered by the source table's row policy, so autocomplete can surface Map key names from rows the user cannot read. This does not trigger on the shipped OTel schema (its rollups do not index Map columns, so they return empty and fall through to the policy-filtered scan) and leaks key names only, so it stays P2.
    • Fix: Document the requirement to mirror source row policies onto custom Map-indexing rollup tables, and consider skipping the rollup for any text-index failure (not only access-denied) when a row policy may apply.
    • security, adversarial, previous-comments
🔵 P3 nitpicks (3)
  • packages/common-utils/src/core/metadata.ts:1107 — When a non-access-denied text-index read fails and the rollup returns keys, the result is cached under the .rollup key but not under cacheKey, so each later call re-issues the failing text-index query and re-logs the warning before hitting the rollup cache; the existing cache test only exercises the scan-cache path, not the rollup-success path.
    • Fix: Cache the successful rollup result under cacheKey as well, and add a test asserting the failing index query is not re-issued on the rollup-success path.
  • packages/common-utils/src/core/metadata.ts:1008 — The key and kv text-index branches are near-identical ~24-line try/catch/query/cache blocks (also flagged by the duplicate-code check); the abort/row-policy/cache logic must be maintained in two places.
    • Fix: Extract a private helper parameterized by index name, the key SELECT expression, and the warn label.
    • maintainability
  • packages/common-utils/src/core/metadata.ts:996 — The comment "Text Index path: query the key rollup index" reuses "rollup," which also names the distinct keyRollupTable path below, inviting confusion between the two mechanisms.
    • Fix: Reword to reference mergeTreeTextIndex so the text-index path is distinct from the rollup-table path.
    • maintainability

Reviewers (11): correctness, reliability, security, performance, testing, maintainability, kieran-typescript, adversarial, previous-comments, project-standards, learnings.

Testing gaps: The rollup-success-after-failure retry/caching path (non-access-denied failure where the rollup returns keys) is not covered; add a case asserting the failing index query is not re-issued on later calls through that path.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

E2E Test Results

✅ All tests passed • 392 passed • 1 skipped • 1629s

Status Count
✅ Passed 392
❌ Failed 0
⚠️ Flaky 2
⏭️ Skipped 1

Tests ran across 4 shards in parallel.

View full report →

…d text index reads

Rollup tables are separate tables that a row policy on the source table
does not cover. Once mergeTreeTextIndex fails with ACCESS_DENIED we know a
policy applies, so skip the rollup and read keys only from the
policy-filtered source-table scan. Other text index failures still try the
rollup first.

Also cap the text index read at 15s (break on overflow, matching the other
getMapKeys strategies) and forward the abort signal, so a slow index read
plus the scan fallback can't stack up.

Refs HDX-5598
} catch (e) {
// e.g. ACCESS_DENIED under a row policy or BAD_ARGUMENTS on a
// Distributed table; the paths below still work.
rowPolicyDenied = isAccessDeniedError(e);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Aborting during the text index read now falls through and caches an empty rollup result with no TTL

Before this PR the text index query didn't get abort_signal, so it couldn't be cancelled. Now it does, and both catch blocks treat a cancelled request as an ordinary failure. The call then moves on to the rollup query with a signal that is already aborted. The rollup getOrFetch catches that error and caches [] under its .rollup key (metadata.ts:1102-1156). That cache has no TTL, so later non-aborted calls skip a custom Map-indexing rollup for that range and always run the source scan. Every cancellation also logs a misleading "falling through" warning. describeSource.ts:240 passes signal today, so this path is reachable. Fix: in both catches, check for abort before setting rowPolicyDenied. If signal?.aborted, rethrow, and add a test for that case.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

timeout_overflow_mode 'break' would return a partial key list as a success
and write it to the TTL-less metadata cache, and the pinned
max_execution_time overrode the deployment's own setting. Planning-time
failures like ACCESS_DENIED don't need it. Keep forwarding the abort signal.

Refs HDX-5598
…th throw-on-timeout

An aborted text index read used to fall through to the rollup with a dead
signal; the rollup rejected at once, returned [], and getOrFetch cached
that under the rollup key with no TTL, disabling the rollup for that
column and range for the life of the process (reachable from the MCP
describe_source timeout). Rethrow when the signal is aborted.

Cap the index read at 15s with timeout_overflow_mode 'throw', matching the
rollup and scan caps, so a slow read falls through promptly. 'throw'
rather than 'break' so a timeout never caches a partial key list.

Refs HDX-5598
@kodiakhq
kodiakhq Bot merged commit 0b85553 into main Oct 2, 2026
29 checks passed
@kodiakhq
kodiakhq Bot deleted the warren/HDX-5598-map-keys-text-index-fallthrough branch October 2, 2026 09:45

This branch was successfully deployed

2 active deployments
Preview – hyperdx-oss — fbd16b3c Deployed Oct 2, 2026 by vercel[bot]
Preview – hyperdx-storybook — fbd16b3c Deployed Oct 2, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-generated AI-generated content; review carefully before merging. automerge review/tier-2 Low risk — AI review + quick human skim

Projects

None yet

Development

Successfully merging this pull request may close these issues.

getMapKeys returns no keys when the mergeTreeTextIndex query fails (e.g. ACCESS_DENIED under row policies) instead of falling through

2 participants