Repository navigation
fix(app): let the Database query drawer use the span attribute skip index #3351
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
1f51945
35235fc
46cb39b
d60acf2
474f80a
f5b4b4e
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| --- | ||
| '@hyperdx/app': patch | ||
| --- | ||
|
|
||
| fix: Let the Services > Database query drawer use the span attribute skip index instead of scanning the whole trace window |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,12 +1,16 @@ | ||
| import { useMemo } from 'react'; | ||
| import { ColumnMeta } from '@hyperdx/common-utils/dist/clickhouse'; | ||
| import { tcFromSource } from '@hyperdx/common-utils/dist/core/metadata'; | ||
| import { escapeSqlString } from '@hyperdx/common-utils/dist/core/utils'; | ||
| import { SourceKind, TTraceSource } from '@hyperdx/common-utils/dist/types'; | ||
|
|
||
| import { useColumns, useJsonColumns } from './hooks/useMetadata'; | ||
|
|
||
| const COALESCE_FIELDS_LIMIT = 100; | ||
|
|
||
| // Ordered from highest to lowest precedence | ||
| const DB_STATEMENT_KEYS = ['db.query.text', 'db.statement']; | ||
|
|
||
| // Helper function to format field access based on column type | ||
| function formatFieldAccess( | ||
| field: string, | ||
|
|
@@ -89,21 +93,21 @@ function getDefaults({ | |
| isAttributeFieldJSON?: boolean; | ||
| } = {}) { | ||
| const dbStatement = makeCoalescedFieldsAccessQuery( | ||
| [ | ||
| formatFieldAccess( | ||
| spanAttributeField, | ||
| 'db.query.text', | ||
| isAttributeFieldJSON, | ||
| ), | ||
| formatFieldAccess( | ||
| spanAttributeField, | ||
| 'db.statement', | ||
| isAttributeFieldJSON, | ||
| ), | ||
| ], | ||
| DB_STATEMENT_KEYS.map(key => | ||
| formatFieldAccess(spanAttributeField, key, isAttributeFieldJSON), | ||
| ), | ||
| isAttributeFieldJSON, | ||
| ); | ||
|
|
||
| // Plain map accessors: the shape `rewriteSqlFilterWithKvItems` rewrites onto | ||
| // an indexed items column. JSON columns have no such column, and their | ||
| // accessors are not map subscripts, so they get no prefilter. | ||
| const dbStatementPrefilterFields = isAttributeFieldJSON | ||
| ? [] | ||
| : DB_STATEMENT_KEYS.map(key => | ||
| formatFieldAccess(spanAttributeField, key, false), | ||
| ); | ||
|
|
||
| return { | ||
| duration: 'Duration', | ||
| durationPrecision: 9, | ||
|
|
@@ -133,11 +137,45 @@ function getDefaults({ | |
| isAttributeFieldJSON, | ||
| ), | ||
| dbStatement, | ||
| dbStatementPrefilterFields, | ||
| }; | ||
| } | ||
|
|
||
| const ENDPOINT_MATERIALIZED_COLUMN_NAME = 'endpoint'; | ||
|
|
||
| export function inList(expression: string, value: string): string { | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 minor —
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Sharing it would mean moving the helper into
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 minor — The sibling Endpoint drawer still interpolates endpoint and service without escaping
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Pre-existing, and in a file this PR does not touch, so I am leaving it. It is worth its own fix though: an endpoint or service name containing a quote breaks every query in that drawer. |
||
| return `${expression} IN ('${escapeSqlString(value)}')`; | ||
| } | ||
|
|
||
| /** | ||
| * Condition selecting the spans of a single database statement. | ||
| * | ||
| * No skip index covers the coalesced attribute lookup, so on its own it makes | ||
| * ClickHouse read every granule in the time range. The prefilter compares the | ||
| * same keys directly, which matches a superset of those spans while the | ||
| * coalesced comparison still decides; `rewriteSqlFilterWithKvItems` then | ||
| * rewrites each comparison onto the schema's indexed items column, and older | ||
| * schemas serve them from their `mapValues` index as written. | ||
| */ | ||
| export function makeDbStatementCondition({ | ||
| expressions, | ||
| statement, | ||
| }: { | ||
| expressions: Pick< | ||
| ReturnType<typeof getExpressions>, | ||
| 'dbStatement' | 'dbStatementPrefilterFields' | ||
| >; | ||
| statement: string; | ||
| }): string { | ||
| const equality = inList(expressions.dbStatement, statement); | ||
| const prefilter = expressions.dbStatementPrefilterFields.map(field => | ||
| inList(field, statement), | ||
| ); | ||
| return prefilter.length | ||
| ? `(${prefilter.join(' OR ')}) AND ${equality}` | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟠 major — The OR prefilter turns into an AND when you drill down to search, so spans disappear
Nothing blocks merge automatically, but a maintainer will expect this fixed if it is a real defect in code this PR changes. If it is about surrounding code, reply and say so instead of patching. Do not widen the PR. How to respond
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I reproduced this scenario in the running product and it does not happen. The drill-down URL the drawer generated, [{"type":"sql","condition":"(SpanAttributes['db.query.text'] IN ('{\"create\":\"?\"}') OR SpanAttributes['db.statement'] IN ('{\"create\":\"?\"}')) AND coalesce(nullif(SpanAttributes['db.query.text'], ''), nullif(SpanAttributes['db.statement'], '')) IN ('{\"create\":\"?\"}')"}]Opening it, the search sidebar renders one facet,
AND ((has(SpanAttributeItems, concat('db.query.text', '=', '{"create":"?"}')) OR has(SpanAttributeItems, concat('db.statement', '=', '{"create":"?"}'))) AND (coalesce(...
AND ((coalesce(nullif(SpanAttributes['db.query.text'], ''), nullif(SpanAttributes['db.statement'], '')) IN ('{"create":"?"}')) AND (ServiceName IN ('hdx-oss-dev-api')))The regenerated condition is the coalesce plus the facet the user added; the two attributes are never ANDed, so no span is dropped. What is lost is the prefilter, which is only an index hint — search falls back to the behaviour it has today. If I am misreading the scenario, point me at the step where a second facet appears. |
||
| : equality; | ||
| } | ||
|
|
||
| export function getExpressions( | ||
| source: TTraceSource, | ||
| columns: ColumnMeta[], | ||
|
|
@@ -172,6 +210,7 @@ export function getExpressions( | |
|
|
||
| // Database | ||
| dbStatement: defaults.dbStatement, | ||
| dbStatementPrefilterFields: defaults.dbStatementPrefilterFields, | ||
| }; | ||
|
|
||
| const auxExpressions = { | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🔵 minor — No test sends a statement with quotes or backslashes through the kv-items rewrite
The rewrite test only uses
'SELECT 1'. With a quote or backslash, the statement passes throughescapeSqlString, then node-sql-parser, thenSqlString.formatre-escaping ingenerateHasSqlForKvItemsColumn(renderChartConfig.ts:383). Other code (renderChartConfig.ts:326) puts parsedsingle_quote_stringvalues back between quotes without re-escaping them, which suggests the parser keeps the SQL escapes. If so,''gets escaped a second time,has(items, …)never matches, and the drawer shows nothing. Run"SELECT 'a' WHERE x LIKE 'a\\_b'"throughrewriteSqlFilterWithKvItemsand check the decoded literal equals the original statement.Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The behaviour holds; the gap is the test. From
system.query_log, a statement full of quotes and braces going through the rewrite and running:I have the test written locally. This is the third round of findings, so I would rather a maintainer says whether to add it than keep pushing commits.