Skip to content

Commit 24767c5

Browse files
authored
fix: Ensure correct bounds for date-based timestampValueExpr (#2066)
## Summary This PR fixes the time filter expression generated when a source's timestamp value expression includes a Date-type column. In such cases, the bounds should be inclusive for the date-type columns. This extends the fix from #1915 to date-type columns. ### Screenshots or video #### Before - Histogram is empty because of exclusive end bounds <img width="1758" height="700" alt="Screenshot 2026-04-07 at 12 32 05 PM" src="https://github.com/user-attachments/assets/95898655-21f1-4380-9b23-5333fcea007f" /> #### After - Histogram is populated <img width="1762" height="717" alt="Screenshot 2026-04-07 at 12 30 25 PM" src="https://github.com/user-attachments/assets/d159b131-1f4f-44f8-982e-b26ca68835ff" /> ### How to test locally or on Vercel The unit tests demonstrate the fix. <details> <summary>Alternatively, create a table with the following schema and data</summary> ```sql CREATE TABLE default.otel_logs_date_pk ( `Timestamp` DateTime64(9) CODEC(Delta(8), ZSTD(1)), `TimestampTime` DateTime DEFAULT toDateTime(Timestamp), `TimestampDate` Date DEFAULT toDate(Timestamp), `TraceId` String CODEC(ZSTD(1)), `SpanId` String CODEC(ZSTD(1)), `TraceFlags` UInt8, `SeverityText` LowCardinality(String) CODEC(ZSTD(1)), `SeverityNumber` UInt8, `ServiceName` LowCardinality(String) CODEC(ZSTD(1)), `Body` String CODEC(ZSTD(1)), `ResourceSchemaUrl` LowCardinality(String) CODEC(ZSTD(1)), `ResourceAttributes` Map(LowCardinality(String), String) CODEC(ZSTD(1)), `ScopeSchemaUrl` LowCardinality(String) CODEC(ZSTD(1)), `ScopeName` String CODEC(ZSTD(1)), `ScopeVersion` LowCardinality(String) CODEC(ZSTD(1)), `ScopeAttributes` Map(LowCardinality(String), String) CODEC(ZSTD(1)), `LogAttributes` Map(LowCardinality(String), String) CODEC(ZSTD(1)), `__hdx_materialized_k8s.cluster.name` LowCardinality(String) MATERIALIZED ResourceAttributes['k8s.cluster.name'] CODEC(ZSTD(1)), `__hdx_materialized_k8s.container.name` LowCardinality(String) MATERIALIZED ResourceAttributes['k8s.container.name'] CODEC(ZSTD(1)), `__hdx_materialized_k8s.deployment.name` LowCardinality(String) MATERIALIZED ResourceAttributes['k8s.deployment.name'] CODEC(ZSTD(1)), `__hdx_materialized_k8s.namespace.name` LowCardinality(String) MATERIALIZED ResourceAttributes['k8s.namespace.name'] CODEC(ZSTD(1)), `__hdx_materialized_k8s.node.name` LowCardinality(String) MATERIALIZED ResourceAttributes['k8s.node.name'] CODEC(ZSTD(1)), `__hdx_materialized_k8s.pod.name` LowCardinality(String) MATERIALIZED ResourceAttributes['k8s.pod.name'] CODEC(ZSTD(1)), `__hdx_materialized_k8s.pod.uid` LowCardinality(String) MATERIALIZED ResourceAttributes['k8s.pod.uid'] CODEC(ZSTD(1)), `__hdx_materialized_deployment.environment.name` LowCardinality(String) MATERIALIZED ResourceAttributes['deployment.environment.name'] CODEC(ZSTD(1)), INDEX idx_trace_id TraceId TYPE bloom_filter(0.001) GRANULARITY 1, INDEX idx_res_attr_key mapKeys(ResourceAttributes) TYPE bloom_filter(0.01) GRANULARITY 1, INDEX idx_res_attr_value mapValues(ResourceAttributes) TYPE bloom_filter(0.01) GRANULARITY 1, INDEX idx_scope_attr_key mapKeys(ScopeAttributes) TYPE bloom_filter(0.01) GRANULARITY 1, INDEX idx_scope_attr_value mapValues(ScopeAttributes) TYPE bloom_filter(0.01) GRANULARITY 1, INDEX idx_log_attr_key mapKeys(LogAttributes) TYPE bloom_filter(0.01) GRANULARITY 1, INDEX idx_log_attr_value mapValues(LogAttributes) TYPE bloom_filter(0.01) GRANULARITY 1, INDEX idx_lower_body lower(Body) TYPE tokenbf_v1(32768, 3, 0) GRANULARITY 8 ) ENGINE = MergeTree PARTITION BY toDate(TimestampTime) PRIMARY KEY (TimestampDate, ServiceName, TimestampTime) ORDER BY (TimestampDate, ServiceName, TimestampTime, Timestamp) TTL TimestampTime + toIntervalDay(1) SETTINGS index_granularity = 8192, ttl_only_drop_parts = 1; insert into default.otel_logs_date_pk (Timestamp, SeverityText, Body) VALUES (now(), 'info', 'message'); insert into default.otel_logs_date_pk (Timestamp, SeverityText, Body) VALUES (now()-interval 1 minute, 'info', 'message'); ``` Then create a source that points to that table with the timestampValueExpr `TimestampTime, TimestampDate` </details> ### References - Linear Issue: Closes HDX-3930 - Related PRs:
1 parent ad71dc2 commit 24767c5

3 files changed

Lines changed: 99 additions & 7 deletions

File tree

‎.changeset/hot-camels-sing.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
"@hyperdx/common-utils": patch
3+
---
4+
5+
fix: Ensure correct bounds for date-based timestampValueExpr

‎packages/common-utils/src/__tests__/renderChartConfig.test.ts‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1293,6 +1293,52 @@ describe('renderChartConfig', () => {
12931293
dateRangeStartInclusive: false,
12941294
expected: `(toStartOfHour(timestamp) >= toStartOfHour(fromUnixTimestamp64Milli(${new Date('2025-02-12 03:53:38Z').getTime()})) AND toStartOfHour(timestamp) <= toStartOfHour(fromUnixTimestamp64Milli(${new Date('2025-02-12 04:08:38Z').getTime()})))`,
12951295
},
1296+
{
1297+
description: 'stays inclusive with date-type column',
1298+
timestampValueExpression: 'date',
1299+
dateRange: [
1300+
new Date('2025-02-12 03:53:38Z'),
1301+
new Date('2025-02-12 04:08:38Z'),
1302+
],
1303+
dateRangeStartInclusive: false,
1304+
dateRangeEndInclusive: false,
1305+
expected: `(date >= toDate(fromUnixTimestamp64Milli(${new Date('2025-02-12 03:53:38Z').getTime()})) AND date <= toDate(fromUnixTimestamp64Milli(${new Date('2025-02-12 04:08:38Z').getTime()})))`,
1306+
},
1307+
{
1308+
description:
1309+
'stays inclusive for date-type column in multi-column timestampValueExpression',
1310+
timestampValueExpression: 'date, timestamp',
1311+
dateRange: [
1312+
new Date('2025-02-12 03:53:38Z'),
1313+
new Date('2025-02-12 04:08:38Z'),
1314+
],
1315+
dateRangeStartInclusive: false,
1316+
dateRangeEndInclusive: false,
1317+
expected: `(date >= toDate(fromUnixTimestamp64Milli(${new Date('2025-02-12 03:53:38Z').getTime()})) AND date <= toDate(fromUnixTimestamp64Milli(${new Date('2025-02-12 04:08:38Z').getTime()})))AND(timestamp > fromUnixTimestamp64Milli(${new Date('2025-02-12 03:53:38Z').getTime()}) AND timestamp < fromUnixTimestamp64Milli(${new Date('2025-02-12 04:08:38Z').getTime()}))`,
1318+
},
1319+
{
1320+
description: 'stays inclusive for toDate column',
1321+
timestampValueExpression: 'toDate(timestamp)',
1322+
dateRange: [
1323+
new Date('2025-02-12 03:53:38Z'),
1324+
new Date('2025-02-12 04:08:38Z'),
1325+
],
1326+
dateRangeStartInclusive: false,
1327+
dateRangeEndInclusive: false,
1328+
expected: `(toDate(timestamp) >= toDate(fromUnixTimestamp64Milli(${new Date('2025-02-12 03:53:38Z').getTime()})) AND toDate(timestamp) <= toDate(fromUnixTimestamp64Milli(${new Date('2025-02-12 04:08:38Z').getTime()})))`,
1329+
},
1330+
{
1331+
description:
1332+
'stays inclusive for toDate column in multi-column timestampValueExpression',
1333+
timestampValueExpression: 'toDate(timestamp), timestamp',
1334+
dateRange: [
1335+
new Date('2025-02-12 03:53:38Z'),
1336+
new Date('2025-02-12 04:08:38Z'),
1337+
],
1338+
dateRangeStartInclusive: false,
1339+
dateRangeEndInclusive: false,
1340+
expected: `(toDate(timestamp) >= toDate(fromUnixTimestamp64Milli(${new Date('2025-02-12 03:53:38Z').getTime()})) AND toDate(timestamp) <= toDate(fromUnixTimestamp64Milli(${new Date('2025-02-12 04:08:38Z').getTime()})))AND(timestamp > fromUnixTimestamp64Milli(${new Date('2025-02-12 03:53:38Z').getTime()}) AND timestamp < fromUnixTimestamp64Milli(${new Date('2025-02-12 04:08:38Z').getTime()}))`,
1341+
},
12961342
];
12971343

12981344
beforeEach(() => {
@@ -1338,6 +1384,36 @@ describe('renderChartConfig', () => {
13381384
expect(actualSql).toBe(expected);
13391385
},
13401386
);
1387+
1388+
it('stays inclusive for date-type column with non-subquery with clauses', async () => {
1389+
const dateRange: [Date, Date] = [
1390+
new Date('2025-02-12 03:53:38Z'),
1391+
new Date('2025-02-12 04:08:38Z'),
1392+
];
1393+
1394+
const actual = await timeFilterExpr({
1395+
timestampValueExpression: 'date',
1396+
dateRangeEndInclusive: false,
1397+
dateRangeStartInclusive: false,
1398+
dateRange,
1399+
connectionId: 'test-connection',
1400+
databaseName: 'default',
1401+
tableName: 'target_table',
1402+
metadata: mockMetadata,
1403+
with: [
1404+
{
1405+
name: 'service',
1406+
sql: { sql: 'ServiceName', params: {} },
1407+
isSubquery: false,
1408+
},
1409+
],
1410+
});
1411+
1412+
const actualSql = parameterizedQueryToSql(actual);
1413+
expect(actualSql).toBe(
1414+
`(date >= toDate(fromUnixTimestamp64Milli(${dateRange[0].getTime()})) AND date <= toDate(fromUnixTimestamp64Milli(${dateRange[1].getTime()})))`,
1415+
);
1416+
});
13411417
});
13421418

13431419
it('should not generate invalid SQL when primary key wraps toStartOfInterval', async () => {

‎packages/common-utils/src/core/renderChartConfig.ts‎

Lines changed: 18 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -686,8 +686,11 @@ export async function timeFilterExpr({
686686
// timestamp comparison must also have the same function
687687
const toStartOf = parseToStartOfFunction(col);
688688

689+
// Detect toDate(...) wrapper expressions
690+
const isToDateExpr = /^toDate\s*\(/.test(col);
691+
689692
const columnMeta =
690-
withClauses?.length || toStartOf
693+
hasSubqueryCte(withClauses) || toStartOf || isToDateExpr
691694
? null
692695
: await metadata.getColumn({
693696
databaseName,
@@ -700,7 +703,12 @@ export async function timeFilterExpr({
700703
UNSAFE_RAW_SQL: col,
701704
};
702705

703-
if (columnMeta == null && !withClauses?.length && !toStartOf) {
706+
if (
707+
columnMeta == null &&
708+
hasSubqueryCte(withClauses) &&
709+
!toStartOf &&
710+
!isToDateExpr
711+
) {
704712
console.warn(
705713
`Column ${col} not found in ${databaseName}.${tableName} while inferring type for time filter`,
706714
);
@@ -718,12 +726,15 @@ export async function timeFilterExpr({
718726
? chSql`${toStartOf.function}(fromUnixTimestamp64Milli(${{ Int64: endTime }})${toStartOf.formattedRemainingArgs})`
719727
: chSql`fromUnixTimestamp64Milli(${{ Int64: endTime }})`;
720728

721-
// toStartOf* filters must stay inclusive — strict < on a rounded value drops a whole interval
722-
const startOp = dateRangeStartInclusive || toStartOf ? '>=' : '>';
723-
const endOp = dateRangeEndInclusive || toStartOf ? '<=' : '<';
729+
const isDateType = columnMeta?.type === 'Date' || isToDateExpr;
730+
731+
// toStartOf* and Date filters must stay inclusive — strict < on a rounded value drops a whole interval
732+
const startOp =
733+
dateRangeStartInclusive || toStartOf || isDateType ? '>=' : '>';
734+
const endOp =
735+
dateRangeEndInclusive || toStartOf || isDateType ? '<=' : '<';
724736

725-
// If it's a date type
726-
if (columnMeta?.type === 'Date') {
737+
if (isDateType) {
727738
return chSql`(${unsafeTimestampValueExpression} ${startOp} toDate(${startTimeCond}) AND ${unsafeTimestampValueExpression} ${endOp} toDate(${endTimeCond}))`;
728739
} else {
729740
return chSql`(${unsafeTimestampValueExpression} ${startOp} ${startTimeCond} AND ${unsafeTimestampValueExpression} ${endOp} ${endTimeCond})`;

0 commit comments

Comments
 (0)