Repository navigation
fix(app): let the Database query drawer use the span attribute skip index - #3351
kodiakhq[bot] merged 6 commits into
Conversation
…ndex
Opening a database statement filtered on the coalesced attribute lookup alone.
No skip index covers that expression, so every tile in the drawer read each
granule in the selected window, which is what times out on a busy source.
Prefilter with an expression an index does cover, in the form the table
supports: `<Name>AttributeItems` where the schema has that column, otherwise
`mapValues(<Name>Attributes)`. The prefilter matches a superset of the spans
and the coalesced comparison still decides, so results are unchanged. It also
helps where neither is indexed, by short-circuiting the coalesce.
Measured on 4M spans, one statement, caches off:
granules rows read bytes CPU
before 499/499 4,001,873 191.8MiB 2945ms
after 26/499 212,992 22.4MiB 256ms
The older schema reached the same ratios through its mapValues index. Both
were confirmed end to end against ClickHouse 26.8 and 26.1.
JSON attribute columns keep the previous condition, and statements are now
escaped, so one containing a quote no longer breaks the query.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
🦋 Changeset detectedLatest commit: 474f80a The changes in this PR will be included in the next version bump. This PR includes changesets to release 3 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
@archat-hash is attempting to deploy a commit to the HyperDX Team on Vercel. A member of the Team first needs to authorize it. |
| return undefined; | ||
| } | ||
|
|
||
| const itemsColumn = getAttributeItemsColumn(attributesField, columns); |
There was a problem hiding this comment.
🟠 major — items prefilter skips the version, column-kind and index checks that the existing KV-items rewrite applies
The items branch is picked whenever a column named <X>AttributeItems exists. It does not check the server version, whether the column is an ALIAS/MATERIALIZED key=value column, or whether it has a text(tokenizer='array') index. populateValidKvTextIndices in packages/common-utils/src/queryParser.ts:1239 does check these, and supportsDirectReadMap in core/clickhouseVersion.ts:121 deliberately turns has(<ALIAS items>, …) off on 26.2 < 26.2.19.43, 26.3 < 26.3.12.3 and 26.4 < 26.4.3.37. On those versions an ALIAS items column has no direct_read, so ClickHouse recomputes the arrayMap for every row. The stock traces schema can exist there, since 26.2 can create it, but the PR only measured 26.8 and 26.1. The separator is also hardcoded as = instead of being parsed from the column expression. Fix: drop getAttributeItemsColumn and the items kind. Emit the prefilter as (SpanAttributes['db.query.text'] = '…' OR SpanAttributes['db.statement'] = '…') and let rewriteSqlFilterWithKvItems (core/renderChartConfig.ts:391, already run on every sql filter in renderWhere) turn it into has(items, concat(…)) only when the lookup allows it. That also picks up hasAny on 26.5+. Keep the mapValues form only for schemas without a KV items index.
Nothing blocks merge automatically, but a maintainer will expect this fixed if it is a real defect in code this PR changes. If it is about surrounding code, reply and say so instead of patching. Do not widen the PR. How to respond
There was a problem hiding this comment.
Fixed in 35235fc. The app no longer picks the column: it emits plain map comparisons and rewriteSqlFilterWithKvItems does the rewrite, so the direct_read/version check, the ALIAS-vs-MATERIALIZED test and the text-index match all come from populateValidKvTextIndices instead of being duplicated here. Confirmed on 26.8 (drawer queries arrive as has(SpanAttributeItems, concat(...)), 9 of 492 granules) and on 26.1, where the lookup is empty and the comparisons reach ClickHouse as written and are served by idx_span_attr_value (57,310 rows read instead of 4,000,562).
| filters.push({ | ||
| type: 'sql', | ||
| condition: `${expressions?.service} IN ('${service}')`, | ||
| condition: `${expressions.service} IN ('${service}')`, |
There was a problem hiding this comment.
🔵 minor — Service filter is still built from an unescaped string
This PR escapes the statement, but the service condition it also edits still interpolates service raw. A service name containing ' or \ produces broken SQL, and every tile in the drawer errors. Wrap it as '${escapeSqlString(service)}' (from @hyperdx/common-utils/dist/core/utils), as makeDbStatementCondition does.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
There was a problem hiding this comment.
Fixed in 35235fc — wrapped in escapeSqlString, same as the statement.
PR Review
4 finding(s): 🔴 0 critical · 🟠 2 major · 🔵 2 minor 2 posted as inline comment(s) on the changed lines. 2 listed below. Findings outside the changed lines
Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides. |
|
Deep ReviewScope: The change is sound. The prefilter is a verified superset of the original coalesced comparison (whenever 🟡 P2 — recommended
🔵 P3 nitpicks (2)
Reviewers (10): correctness, security, performance, testing, maintainability, project-standards, kieran-typescript, previous-comments, agent-native, learnings-researcher. Testing gaps:
No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond |
knudtty
left a comment
There was a problem hiding this comment.
Address the bigger review issues before merging
Review feedback: picking the items column by name skipped the checks the
existing KV-items path already makes — server version and direct_read support,
ALIAS vs MATERIALIZED, a matching text index — and hardcoded `=` as the
separator, so a column built with another separator silently matched nothing.
Emit the prefilter as plain map comparisons instead. `rewriteSqlFilterWithKvItems`
already runs over every sql filter and rewrites that shape onto the validated
items column with the separator read from its DDL, and schemas without such a
column keep the comparisons, which their mapValues index serves. The hint type
and the three helpers that sniffed the schema are gone.
Also escape the service name, which the drawer still interpolated raw.
Confirmed against ClickHouse 26.8: the drawer's queries arrive rewritten to
`has(SpanAttributeItems, concat('db.query.text', '=', …))` and read 9 of 492
granules.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
| filters.push({ | ||
| type: 'sql', | ||
| condition: `${expressions?.service} IN ('${service}')`, | ||
| condition: `${expressions.service} IN ('${escapeSqlString(service)}')`, |
There was a problem hiding this comment.
🔵 minor — The escaped IN ('…') builder is now written twice: inList and an inline copy here
Export inList from serviceDashboard.ts and use it for the service filter. Then quoting and escaping are defined in one place. ServiceDashboardEndpointSidePanel.tsx:59,65 could also use it, since it still interpolates endpoint and service without escaping.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
There was a problem hiding this comment.
Fixed in the latest commit: inList is exported and the service filter uses it, so quoting and escaping live in one place. I left ServiceDashboardEndpointSidePanel.tsx alone since this PR does not touch that file.
Review feedback: the escaped `IN ('…')` was spelled out twice after the last
commit. Export the helper and use it for the service filter too, so quoting and
escaping stay in one place.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
| inList(field, statement), | ||
| ); | ||
| return prefilter.length | ||
| ? `(${prefilter.join(' OR ')}) AND ${equality}` |
There was a problem hiding this comment.
🟠 major — The OR prefilter turns into an AND when you drill down to search, so spans disappear
parseQuery (packages/common-utils/src/filters.ts:532, extractInClauses) takes every IN clause in the condition, so a drawer chart's drill-down (ChartUtils.tsx:1387 serializes config.filters into the search URL) gives the sidebar three facets: SpanAttributes['db.query.text'], SpanAttributes['db.statement'] and the coalesce. The first time someone changes a filter, updateFilterQuery (searchFilters.tsx:137) rebuilds the query with filtersToQuery, which ANDs those facets together. Any span that has only one of the two attributes then gets dropped, and that is the usual case. Keep the prefilter out of the user-visible filter: add it at render time (e.g. as indexHint(...) or an internal condition the renderer adds), or make parseQuery ignore it. Also make the round-trip test assert Object.keys(filters) equals [dbStatement], not just that filters[dbStatement] is defined.
Nothing blocks merge automatically, but a maintainer will expect this fixed if it is a real defect in code this PR changes. If it is about surrounding code, reply and say so instead of patching. Do not widen the PR. How to respond
There was a problem hiding this comment.
I reproduced this scenario in the running product and it does not happen.
The drill-down URL the drawer generated, filters param decoded:
[{"type":"sql","condition":"(SpanAttributes['db.query.text'] IN ('{\"create\":\"?\"}') OR SpanAttributes['db.statement'] IN ('{\"create\":\"?\"}')) AND coalesce(nullif(SpanAttributes['db.query.text'], ''), nullif(SpanAttributes['db.statement'], '')) IN ('{\"create\":\"?\"}')"}]Opening it, the search sidebar renders one facet, coalesce(nullif(...)) — not three. The parenthesised OR never becomes one: extractInClauses skips it at filters.ts:431 (if (containsOperatorOutsideQuotes(part)) continue;).
system.query_log before touching any filter, 12 results:
AND ((has(SpanAttributeItems, concat('db.query.text', '=', '{"create":"?"}')) OR has(SpanAttributeItems, concat('db.statement', '=', '{"create":"?"}'))) AND (coalesce(...system.query_log right after checking a ServiceName value in the sidebar — the updateFilterQuery → filtersToQuery path you describe — still 12 results:
AND ((coalesce(nullif(SpanAttributes['db.query.text'], ''), nullif(SpanAttributes['db.statement'], '')) IN ('{"create":"?"}')) AND (ServiceName IN ('hdx-oss-dev-api')))The regenerated condition is the coalesce plus the facet the user added; the two attributes are never ANDed, so no span is dropped. What is lost is the prefilter, which is only an index hint — search falls back to the behaviour it has today. If I am misreading the scenario, point me at the step where a second facet appears.
| const condition = makeDbStatementCondition({ | ||
| expressions: { dbStatement, dbStatementPrefilterFields: prefilterFields }, | ||
| statement: "SELECT 'a'", | ||
| }); |
There was a problem hiding this comment.
🔵 minor — No test sends a statement with quotes or backslashes through the kv-items rewrite
The rewrite test only uses 'SELECT 1'. With a quote or backslash, the statement passes through escapeSqlString, then node-sql-parser, then SqlString.format re-escaping in generateHasSqlForKvItemsColumn (renderChartConfig.ts:383). Other code (renderChartConfig.ts:326) puts parsed single_quote_string values back between quotes without re-escaping them, which suggests the parser keeps the SQL escapes. If so, '' gets escaped a second time, has(items, …) never matches, and the drawer shows nothing. Run "SELECT 'a' WHERE x LIKE 'a\\_b'" through rewriteSqlFilterWithKvItems and check the decoded literal equals the original statement.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
There was a problem hiding this comment.
The behaviour holds; the gap is the test. From system.query_log, a statement full of quotes and braces going through the rewrite and running:
AND ((has(SpanAttributeItems, concat('db.query.text', '=', '{"update":"?","updates":[{"q":{"_id":"?"},"u":{"$set":{"expires":{}}}}],"ordered":"?"}')) OR has(...I have the test written locally. This is the third round of findings, so I would rather a maintainer says whether to add it than keep pushing commits.
|
|
||
| const ENDPOINT_MATERIALIZED_COLUMN_NAME = 'endpoint'; | ||
|
|
||
| /** A quoted, escaped `IN` comparison — the only place either is spelled out. */ |
There was a problem hiding this comment.
🔵 minor — inList says it is the only place the IN comparison is written out, but the sibling panel still builds it unescaped
ServiceDashboardEndpointSidePanel.tsx:59 and :65 still build ${expressions.endpoint} IN ('${endpoint}') and ${expressions.service} IN ('${service}') with no escaping, so an endpoint or service name containing ' breaks the query. Switch those to inList, or drop the claim from the doc comment. Prior art: filtersToQuery in packages/common-utils/src/filters.ts:75-87 already builds escaped IN ('…') lists, so this helper belongs there or in common-utils rather than in serviceDashboard.ts.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
There was a problem hiding this comment.
You are right, and I apologise for the inaccuracy — the wording overstates it, since ServiceDashboardEndpointSidePanel.tsx still interpolates its values inline. I have the reworded comment locally and will push it if the maintainer wants it in this PR; I did not want to widen the change to that file.
Review feedback: no test drove a statement with quotes or backslashes through rewriteSqlFilterWithKvItems, and the round-trip assertion allowed extra facets instead of pinning the single one the search sidebar gets. Drop the comment on inList, which claimed the IN comparison is spelled out nowhere else while a sibling panel still interpolates its own. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
|
||
| const ENDPOINT_MATERIALIZED_COLUMN_NAME = 'endpoint'; | ||
|
|
||
| export function inList(expression: string, value: string): string { |
There was a problem hiding this comment.
🔵 minor — inList duplicates the escaped IN (...) builder already in filtersToQuery
filtersToQuery (packages/common-utils/src/filters.ts:75-88) already renders ${key} IN ('${escapeSqlString(v)}') for included values. Build the condition from that function, or move one shared single-value helper into common-utils/src/filters.ts that filtersToQuery also uses, so the escaping rules live in one place (the repository's REQUIRED DRY rule).
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
There was a problem hiding this comment.
Sharing it would mean moving the helper into common-utils/src/filters.ts so filtersToQuery uses it too, which is outside the files this PR touches. Flagging rather than widening the change here.
|
|
||
| const ENDPOINT_MATERIALIZED_COLUMN_NAME = 'endpoint'; | ||
|
|
||
| export function inList(expression: string, value: string): string { |
There was a problem hiding this comment.
🔵 minor — The sibling Endpoint drawer still interpolates endpoint and service without escaping
inList is now exported as the shared helper, but packages/app/src/components/ServiceDashboardEndpointSidePanel.tsx:59 and :65 still build IN ('${endpoint}') and IN ('${service}') from raw values. A service or endpoint name containing ' breaks that query. Switch both to inList.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
There was a problem hiding this comment.
Pre-existing, and in a file this PR does not touch, so I am leaving it. It is worth its own fix though: an endpoint or service name containing a quote breaks every query in that drawer.
|
@knudtty everything substantive is addressed and the three review checks pass. Rounds one and two are in The What is left are three advisory notes I deliberately did not act on: two would move code into Since |
Fixes #3034. Follow-up to my comment there; opening this so the reasoning and the numbers are reviewable even if the direction turns out to be different.
The problem
Every tile in the Services → Database query drawer filters by the coalesced statement lookup:
EXPLAIN indexes = 1selects no skip index for it, so each tile reads every granule in the window. The p95 tile is the one that reaches the 60s timeout, but all eight queries a click fires scan the same way.The fix
Prefilter with an expression an index does cover, picked from the table's own columns:
has(<Name>AttributeItems, 'db.query.text=…') OR has(…, 'db.statement=…')where the schema has that column (ClickHouse >= 26.2);has(mapValues(<Name>Attributes), '…'), the map-value index the issue refers to.The prefilter matches a superset of the spans and the coalesced comparison still decides, so results are unchanged. JSON attribute columns and non-map expressions keep the previous condition.
Measured
4M spans, one statement, query and condition caches off:
Same ratios at 20M spans (2477 → 113 granules). Verified end to end on ClickHouse 26.8 and 26.1, identical p95 and row counts in every pair.
Why this shape, and what I rejected
SpanAttributes['db.query.text']directly: loses spans that still carry the olderdb.statement, and on the newer schema it only reaches the key index (1983/2164 granules, 8%).ServiceName/SpanName, which the table's ORDER BY does cover: reaches the same 113 granules, but the top-N query has to collect and pass those values, costing it ~10% CPU and ~3% bytes for no gain over the prefilter.With all skip indexes disabled the prefilter still cuts CPU about 3x by short-circuiting the coalesce before it runs per row, so it does not punish schemas where nothing matches.
Limits
The gain is pruning, so a statement present in most granules gains little.
has(mapValues(…))also matches a span whose unrelated attribute equals the statement text — harmless, the coalesced comparison still decides, it only prunes less.Not addressed
The Database tab's own three queries (top-N and the two per-query charts) still scan the window. They rank across all statements, so there is nothing to prune by; that needs pre-aggregation and did not belong in this PR.
Tests
Both prefilter forms, the fallbacks, escaping, and that the search page's filter parser still reads the condition back as the same filter.
yarn ci:unitinpackages/apppasses except three suites that fail onmaintoo (DBEditTimeChartForm,DashboardFiltersModal,MetricNameSelectSearch).One note on the dev setup
docker/clickhouse/local/config.xmldeclares aprometheus_api_v1HTTP handler, which ClickHouse < 26.2 rejects at startup, so the compat schema path cannot be exercised locally without temporarily removing that block. Flagging in case that is unintentional.🤖 Generated with Claude Code