Skip to content

fix(app): let the Database query drawer use the span attribute skip index - #3351

Merged
kodiakhq[bot] merged 6 commits into
ClickHouse:mainfrom
archat-hash:claude/db-statement-index-prefilter
Oct 13, 2026
Merged

kodiakhq[bot] merged 6 commits into
ClickHouse:mainfrom
archat-hash:claude/db-statement-index-prefilter

Conversation

@archat-hash

Copy link
Copy Markdown
Contributor

Fixes #3034. Follow-up to my comment there; opening this so the reasoning and the numbers are reviewable even if the direction turns out to be different.

The problem

Every tile in the Services → Database query drawer filters by the coalesced statement lookup:

coalesce(nullif(SpanAttributes['db.query.text'], ''), nullif(SpanAttributes['db.statement'], '')) IN ('…')

EXPLAIN indexes = 1 selects no skip index for it, so each tile reads every granule in the window. The p95 tile is the one that reaches the 60s timeout, but all eight queries a click fires scan the same way.

The fix

Prefilter with an expression an index does cover, picked from the table's own columns:

  • has(<Name>AttributeItems, 'db.query.text=…') OR has(…, 'db.statement=…') where the schema has that column (ClickHouse >= 26.2);
  • otherwise has(mapValues(<Name>Attributes), '…'), the map-value index the issue refers to.

The prefilter matches a superset of the spans and the coalesced comparison still decides, so results are unchanged. JSON attribute columns and non-map expressions keep the previous condition.

Measured

4M spans, one statement, query and condition caches off:

granules rows read bytes CPU
before 499/499 4,001,873 191.8 MiB 2945 ms
after 26/499 212,992 22.4 MiB 256 ms

Same ratios at 20M spans (2477 → 113 granules). Verified end to end on ClickHouse 26.8 and 26.1, identical p95 and row counts in every pair.

Why this shape, and what I rejected

  • Dropping the coalesce and comparing SpanAttributes['db.query.text'] directly: loses spans that still carry the older db.statement, and on the newer schema it only reaches the key index (1983/2164 granules, 8%).
  • Adding a non-empty guard: no index covers that either; measured no change (2164/2464 both ways).
  • Narrowing by ServiceName/SpanName, which the table's ORDER BY does cover: reaches the same 113 granules, but the top-N query has to collect and pass those values, costing it ~10% CPU and ~3% bytes for no gain over the prefilter.
  • A materialized column or projection for the statement: strongest asymptotically and what the issue suggests, but a schema change. Happy to go that way instead if you prefer it.

With all skip indexes disabled the prefilter still cuts CPU about 3x by short-circuiting the coalesce before it runs per row, so it does not punish schemas where nothing matches.

Limits

The gain is pruning, so a statement present in most granules gains little. has(mapValues(…)) also matches a span whose unrelated attribute equals the statement text — harmless, the coalesced comparison still decides, it only prunes less.

Not addressed

The Database tab's own three queries (top-N and the two per-query charts) still scan the window. They rank across all statements, so there is nothing to prune by; that needs pre-aggregation and did not belong in this PR.

Tests

Both prefilter forms, the fallbacks, escaping, and that the search page's filter parser still reads the condition back as the same filter. yarn ci:unit in packages/app passes except three suites that fail on main too (DBEditTimeChartForm, DashboardFiltersModal, MetricNameSelectSearch).

One note on the dev setup

docker/clickhouse/local/config.xml declares a prometheus_api_v1 HTTP handler, which ClickHouse < 26.2 rejects at startup, so the compat schema path cannot be exercised locally without temporarily removing that block. Flagging in case that is unintentional.

🤖 Generated with Claude Code

…ndex

Opening a database statement filtered on the coalesced attribute lookup alone.
No skip index covers that expression, so every tile in the drawer read each
granule in the selected window, which is what times out on a busy source.

Prefilter with an expression an index does cover, in the form the table
supports: `<Name>AttributeItems` where the schema has that column, otherwise
`mapValues(<Name>Attributes)`. The prefilter matches a superset of the spans
and the coalesced comparison still decides, so results are unchanged. It also
helps where neither is indexed, by short-circuiting the coalesce.

Measured on 4M spans, one statement, caches off:

              granules   rows read   bytes     CPU
  before      499/499    4,001,873   191.8MiB  2945ms
  after        26/499      212,992    22.4MiB   256ms

The older schema reached the same ratios through its mapValues index. Both
were confirmed end to end against ClickHouse 26.8 and 26.1.

JSON attribute columns keep the previous condition, and statements are now
escaped, so one containing a quote no longer breaks the query.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@changeset-bot

changeset-bot Bot commented Oct 10, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 474f80a

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages
Name Type
@hyperdx/app Patch
@hyperdx/api Patch
@hyperdx/otel-collector Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercel Bot commented Oct 10, 2026

Copy link
Copy Markdown

@archat-hash is attempting to deploy a commit to the HyperDX Team on Vercel.

A member of the Team first needs to authorize it.

Comment thread packages/app/src/serviceDashboard.ts Outdated
return undefined;
}

const itemsColumn = getAttributeItemsColumn(attributesField, columns);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 major — items prefilter skips the version, column-kind and index checks that the existing KV-items rewrite applies

The items branch is picked whenever a column named <X>AttributeItems exists. It does not check the server version, whether the column is an ALIAS/MATERIALIZED key=value column, or whether it has a text(tokenizer='array') index. populateValidKvTextIndices in packages/common-utils/src/queryParser.ts:1239 does check these, and supportsDirectReadMap in core/clickhouseVersion.ts:121 deliberately turns has(<ALIAS items>, …) off on 26.2 < 26.2.19.43, 26.3 < 26.3.12.3 and 26.4 < 26.4.3.37. On those versions an ALIAS items column has no direct_read, so ClickHouse recomputes the arrayMap for every row. The stock traces schema can exist there, since 26.2 can create it, but the PR only measured 26.8 and 26.1. The separator is also hardcoded as = instead of being parsed from the column expression. Fix: drop getAttributeItemsColumn and the items kind. Emit the prefilter as (SpanAttributes['db.query.text'] = '…' OR SpanAttributes['db.statement'] = '…') and let rewriteSqlFilterWithKvItems (core/renderChartConfig.ts:391, already run on every sql filter in renderWhere) turn it into has(items, concat(…)) only when the lookup allows it. That also picks up hasAny on 26.5+. Keep the mapValues form only for schemas without a KV items index.

Nothing blocks merge automatically, but a maintainer will expect this fixed if it is a real defect in code this PR changes. If it is about surrounding code, reply and say so instead of patching. Do not widen the PR. How to respond

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 35235fc. The app no longer picks the column: it emits plain map comparisons and rewriteSqlFilterWithKvItems does the rewrite, so the direct_read/version check, the ALIAS-vs-MATERIALIZED test and the text-index match all come from populateValidKvTextIndices instead of being duplicated here. Confirmed on 26.8 (drawer queries arrive as has(SpanAttributeItems, concat(...)), 9 of 492 granules) and on 26.1, where the lookup is empty and the comparisons reach ClickHouse as written and are served by idx_span_attr_value (57,310 rows read instead of 4,000,562).

filters.push({
type: 'sql',
condition: `${expressions?.service} IN ('${service}')`,
condition: `${expressions.service} IN ('${service}')`,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — Service filter is still built from an unescaped string

This PR escapes the statement, but the service condition it also edits still interpolates service raw. A service name containing ' or \ produces broken SQL, and every tile in the drawer errors. Wrap it as '${escapeSqlString(service)}' (from @hyperdx/common-utils/dist/core/utils), as makeDbStatementCondition does.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 35235fc — wrapped in escapeSqlString, same as the statement.

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

  • 🔴 critical or 🟠 major in code this PR adds or changes: fix before asking for review.
  • 🔵 minor in code this PR changes: your call. Fix if small, otherwise reply.
  • Anything about surrounding code, or asking you to widen the change (hoist a helper,
    dedupe with another file, fix other call sites): never fix here, whatever the
    severity. Reply with one sentence; if it is critical, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

4 finding(s): 🔴 0 critical · 🟠 2 major · 🔵 2 minor

2 posted as inline comment(s) on the changed lines. 2 listed below.

Findings outside the changed lines

  • 🟠 packages/app/src/serviceDashboard.ts:307 — Statements containing ' or \ can get a prefilter that never matches, so the drawer shows no data → inList escapes the value with escapeSqlString (' becomes '', \ becomes \\). As far as I can tell, rewriteSqlFilterWithKvItems (packages/common-utils/src/core/renderChartConfig.ts:446-473) takes the single_quote_string value from node-sql-parser without undoing those escapes, then escapes it again through SqlString.format in generateHasSqlForKvItemsColumn. For SELECT 'a', ClickHouse would then look for has(SpanAttributeItems, concat('db.statement', ':', 'SELECT ''a''')). That never matches, and because the prefilter is ANDed in front of the coalesced comparison, every tile in the drawer would come back empty on schemas that have the items column. Before this PR the coalesce condition was never rewritten, so this path did not exist. Fix: unescape the parsed literal in tryOptimize before building the has(...) call. Until that lands, skip the prefilter when the statement contains ' or \.
  • 🟠 packages/app/src/__tests__/serviceDashboard.test.ts:139 — The escaping test checks quote-count parity, which still passes when the rewritten value is wrong → quoteCount % 2 === 0 and toContain('has(SpanAttributeItems') both pass whether the has() literal decodes to SELECT 'a' FROM t or to SELECT ''a'' FROM t (escaped \' still adds an even number of quotes). Assert the exact has(...) substring for each input, e.g. concat('db.statement', ':', 'SELECT \'a\' FROM t'), so the double-escape above would fail the test.

Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides.

@greptile-apps

greptile-apps Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; no new actionable defect was found.

Summary

Adds direct attribute comparisons to the Database query drawer so the shared renderer can use an indexed items column. The coalesced comparison still selects the final rows.

  • Database drawer queries narrow to the selected statement sooner.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Selected database statement] --> B[Direct attribute comparisons]
  B --> C{Indexed items column available?}
  C -->|Yes| D[Renderer builds has comparisons]
  C -->|No| E[Keep map comparisons]
  D --> F[Apply original coalesced comparison]
  E --> F
  F --> G[Drawer charts and slow queries]
Loading

Reviews (6) · Last reviewed commit: "Merge branch 'main' into claude/db-state..." · Reviewed by Greptile

Comment thread packages/app/src/serviceDashboard.ts Resolved
Comment thread packages/app/src/serviceDashboard.ts Resolved
@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Deep Review

Scope: base:0b9153d5 → HEAD (4 files) · query-builder refactor for the Services → Database query drawer.
Intent: Replace the drawer's un-indexable coalesced statement filter with an indexed prefilter so ClickHouse prunes granules instead of scanning the whole trace window, while routing all value interpolation through escapeSqlString.

The change is sound. The prefilter is a verified superset of the original coalesced comparison (whenever coalesce(nullif(a,''), nullif(b,'')) = v, one of a/b equals v), and the coalesced equality — a function expression the kv-items rewrite never touches — still decides results, so the result set is preserved on the common path. The switch to inList/escapeSqlString also closes a real pre-existing SQL-injection vector where URL-controlled dbquery and service were interpolated raw. All five prior review threads verify as addressed.

🟡 P2 — recommended

  • packages/app/src/serviceDashboard.ts:171 — The new prefilter routes the statement value through the shared rewriteSqlFilterWithKvItems path, which re-parses the ClickHouse-escaped literal under node-sql-parser's Postgresql dialect and re-emits it via SqlString; a statement containing a backslash can be double-escaped so the prefilter has(items, 'key…value') searches a value that never matches the stored one, and because the prefilter is ANDed, the matching span is dropped on kv-indexed schemas. The mechanism and its reachability are pre-existing (this rewrite already runs on every sql-type filter), and the parser-level behavior is unconfirmed, so it is unverified rather than demonstrated.
    • Fix: Add a round-trip test asserting a single-backslash statement produces a rewritten prefilter value equal to the stored value (not just balanced quotes), and normalize backslash handling in the rewrite's value extraction/emission if the test confirms the mismatch.
    • correctness, security
🔵 P3 nitpicks (2)
  • packages/app/src/serviceDashboard.ts:146 — inList(expression, value) renders IN ('<one value>') from a single scalar, so the plural name invites a future caller to pass an array or comma-joined string that escapeSqlString would mis-escape.
    • Fix: Rename to reflect single-value semantics, or widen the signature to accept string[] and escape each element.
    • maintainability, kieran-typescript
  • packages/app/src/serviceDashboard.ts:95 — dbStatement and dbStatementPrefilterFields are built in two independent formatFieldAccess passes with different JSON flags; they stay in sync only via the shared DB_STATEMENT_KEYS constant, so a future change to one accessor path would silently break the prefilter/equality correspondence.
    • Fix: Add a short comment tying the two derivations together, or derive the prefilter fields from the same accessor list the coalesce uses.

Reviewers (10): correctness, security, performance, testing, maintainability, project-standards, kieran-typescript, previous-comments, agent-native, learnings-researcher.

Testing gaps:

  • The escaping-survival test (serviceDashboard.test.ts) asserts only even quote-count and has(...) presence after the rewrite, not that the statement content survives intact — it would not catch the backslash mismatch above.
  • No test exercises the generated SQL when the kv-items lookup is empty (no items column, no mapValues index), where the prefilter degrades to plain map subscripts that prune nothing and add minor per-row cost.
  • No end-to-end assertion that the prefilter+equality result set equals the coalesced-only result set for a span where db.query.text differs but db.statement matches.

No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond

@knudtty knudtty left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good find!

@knudtty knudtty left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Address the bigger review issues before merging

Review feedback: picking the items column by name skipped the checks the
existing KV-items path already makes — server version and direct_read support,
ALIAS vs MATERIALIZED, a matching text index — and hardcoded `=` as the
separator, so a column built with another separator silently matched nothing.

Emit the prefilter as plain map comparisons instead. `rewriteSqlFilterWithKvItems`
already runs over every sql filter and rewrites that shape onto the validated
items column with the separator read from its DDL, and schemas without such a
column keep the comparisons, which their mapValues index serves. The hint type
and the three helpers that sniffed the schema are gone.

Also escape the service name, which the drawer still interpolated raw.

Confirmed against ClickHouse 26.8: the drawer's queries arrive rewritten to
`has(SpanAttributeItems, concat('db.query.text', '=', …))` and read 9 of 492
granules.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
filters.push({
type: 'sql',
condition: `${expressions?.service} IN ('${service}')`,
condition: `${expressions.service} IN ('${escapeSqlString(service)}')`,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — The escaped IN ('…') builder is now written twice: inList and an inline copy here

Export inList from serviceDashboard.ts and use it for the service filter. Then quoting and escaping are defined in one place. ServiceDashboardEndpointSidePanel.tsx:59,65 could also use it, since it still interpolates endpoint and service without escaping.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in the latest commit: inList is exported and the service filter uses it, so quoting and escaping live in one place. I left ServiceDashboardEndpointSidePanel.tsx alone since this PR does not touch that file.

Review feedback: the escaped `IN ('…')` was spelled out twice after the last
commit. Export the helper and use it for the service filter too, so quoting and
escaping stay in one place.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
inList(field, statement),
);
return prefilter.length
? `(${prefilter.join(' OR ')}) AND ${equality}`

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 major — The OR prefilter turns into an AND when you drill down to search, so spans disappear

parseQuery (packages/common-utils/src/filters.ts:532, extractInClauses) takes every IN clause in the condition, so a drawer chart's drill-down (ChartUtils.tsx:1387 serializes config.filters into the search URL) gives the sidebar three facets: SpanAttributes['db.query.text'], SpanAttributes['db.statement'] and the coalesce. The first time someone changes a filter, updateFilterQuery (searchFilters.tsx:137) rebuilds the query with filtersToQuery, which ANDs those facets together. Any span that has only one of the two attributes then gets dropped, and that is the usual case. Keep the prefilter out of the user-visible filter: add it at render time (e.g. as indexHint(...) or an internal condition the renderer adds), or make parseQuery ignore it. Also make the round-trip test assert Object.keys(filters) equals [dbStatement], not just that filters[dbStatement] is defined.

Nothing blocks merge automatically, but a maintainer will expect this fixed if it is a real defect in code this PR changes. If it is about surrounding code, reply and say so instead of patching. Do not widen the PR. How to respond

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reproduced this scenario in the running product and it does not happen.

The drill-down URL the drawer generated, filters param decoded:

[{"type":"sql","condition":"(SpanAttributes['db.query.text'] IN ('{\"create\":\"?\"}') OR SpanAttributes['db.statement'] IN ('{\"create\":\"?\"}')) AND coalesce(nullif(SpanAttributes['db.query.text'], ''), nullif(SpanAttributes['db.statement'], '')) IN ('{\"create\":\"?\"}')"}]

Opening it, the search sidebar renders one facet, coalesce(nullif(...)) — not three. The parenthesised OR never becomes one: extractInClauses skips it at filters.ts:431 (if (containsOperatorOutsideQuotes(part)) continue;).

system.query_log before touching any filter, 12 results:

AND ((has(SpanAttributeItems, concat('db.query.text', '=', '{"create":"?"}')) OR has(SpanAttributeItems, concat('db.statement', '=', '{"create":"?"}'))) AND (coalesce(...

system.query_log right after checking a ServiceName value in the sidebar — the updateFilterQuery → filtersToQuery path you describe — still 12 results:

AND ((coalesce(nullif(SpanAttributes['db.query.text'], ''), nullif(SpanAttributes['db.statement'], '')) IN ('{"create":"?"}')) AND (ServiceName IN ('hdx-oss-dev-api')))

The regenerated condition is the coalesce plus the facet the user added; the two attributes are never ANDed, so no span is dropped. What is lost is the prefilter, which is only an index hint — search falls back to the behaviour it has today. If I am misreading the scenario, point me at the step where a second facet appears.

const condition = makeDbStatementCondition({
expressions: { dbStatement, dbStatementPrefilterFields: prefilterFields },
statement: "SELECT 'a'",
});

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — No test sends a statement with quotes or backslashes through the kv-items rewrite

The rewrite test only uses 'SELECT 1'. With a quote or backslash, the statement passes through escapeSqlString, then node-sql-parser, then SqlString.format re-escaping in generateHasSqlForKvItemsColumn (renderChartConfig.ts:383). Other code (renderChartConfig.ts:326) puts parsed single_quote_string values back between quotes without re-escaping them, which suggests the parser keeps the SQL escapes. If so, '' gets escaped a second time, has(items, …) never matches, and the drawer shows nothing. Run "SELECT 'a' WHERE x LIKE 'a\\_b'" through rewriteSqlFilterWithKvItems and check the decoded literal equals the original statement.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The behaviour holds; the gap is the test. From system.query_log, a statement full of quotes and braces going through the rewrite and running:

AND ((has(SpanAttributeItems, concat('db.query.text', '=', '{"update":"?","updates":[{"q":{"_id":"?"},"u":{"$set":{"expires":{}}}}],"ordered":"?"}')) OR has(...

I have the test written locally. This is the third round of findings, so I would rather a maintainer says whether to add it than keep pushing commits.

Comment thread packages/app/src/serviceDashboard.ts Outdated

const ENDPOINT_MATERIALIZED_COLUMN_NAME = 'endpoint';

/** A quoted, escaped `IN` comparison — the only place either is spelled out. */

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — inList says it is the only place the IN comparison is written out, but the sibling panel still builds it unescaped

ServiceDashboardEndpointSidePanel.tsx:59 and :65 still build ${expressions.endpoint} IN ('${endpoint}') and ${expressions.service} IN ('${service}') with no escaping, so an endpoint or service name containing ' breaks the query. Switch those to inList, or drop the claim from the doc comment. Prior art: filtersToQuery in packages/common-utils/src/filters.ts:75-87 already builds escaped IN ('…') lists, so this helper belongs there or in common-utils rather than in serviceDashboard.ts.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right, and I apologise for the inaccuracy — the wording overstates it, since ServiceDashboardEndpointSidePanel.tsx still interpolates its values inline. I have the reworded comment locally and will push it if the maintainer wants it in this PR; I did not want to widen the change to that file.

Review feedback: no test drove a statement with quotes or backslashes through
rewriteSqlFilterWithKvItems, and the round-trip assertion allowed extra facets
instead of pinning the single one the search sidebar gets.

Drop the comment on inList, which claimed the IN comparison is spelled out
nowhere else while a sibling panel still interpolates its own.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

const ENDPOINT_MATERIALIZED_COLUMN_NAME = 'endpoint';

export function inList(expression: string, value: string): string {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — inList duplicates the escaped IN (...) builder already in filtersToQuery

filtersToQuery (packages/common-utils/src/filters.ts:75-88) already renders ${key} IN ('${escapeSqlString(v)}') for included values. Build the condition from that function, or move one shared single-value helper into common-utils/src/filters.ts that filtersToQuery also uses, so the escaping rules live in one place (the repository's REQUIRED DRY rule).

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sharing it would mean moving the helper into common-utils/src/filters.ts so filtersToQuery uses it too, which is outside the files this PR touches. Flagging rather than widening the change here.


const ENDPOINT_MATERIALIZED_COLUMN_NAME = 'endpoint';

export function inList(expression: string, value: string): string {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 minor — The sibling Endpoint drawer still interpolates endpoint and service without escaping

inList is now exported as the shared helper, but packages/app/src/components/ServiceDashboardEndpointSidePanel.tsx:59 and :65 still build IN ('${endpoint}') and IN ('${service}') from raw values. A service or endpoint name containing ' breaks that query. Switch both to inList.

Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pre-existing, and in a file this PR does not touch, so I am leaving it. It is worth its own fix though: an endpoint or service name containing a quote breaks every query in that drawer.

@archat-hash

Copy link
Copy Markdown
Contributor Author

@knudtty everything substantive is addressed and the three review checks pass.

Rounds one and two are in 35235fc4, 46cb39b5 and d60acf23: the schema sniffing is gone, and the prefilter is emitted in the shape rewriteSqlFilterWithKvItems already rewrites, so the version, column-kind, index and separator checks all come from populateValidKvTextIndices rather than being duplicated. Verified end to end on ClickHouse 26.8, where the drawer's queries arrive as has(SpanAttributeItems, concat(...)) and read 9 of 492 granules, and on 26.1, where the comparisons reach the server as written and idx_span_attr_value serves them — 57,310 rows instead of 4,000,562.

The major in round three I could not reproduce: I opened the drill-down URL the drawer generates and checked a ServiceName facet, and the regenerated condition is the coalesce plus that facet, with the same 12 results before and after. The query_log excerpts are in that thread.

What is left are three advisory notes I deliberately did not act on: two would move code into common-utils or into a sibling panel this PR does not touch, and the third asks to tighten one of my new assertions. Happy to do the last one if you want it here.

Since CHANGES_REQUESTED only clears manually, could you take another look when you have a moment?

@kodiakhq
kodiakhq Bot merged commit c0af58b into ClickHouse:main Oct 13, 2026
26 of 28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Database query details p95 query scans the full trace window and times out

2 participants