Repository navigation
Commit 24767c5
authored
fix: Ensure correct bounds for date-based timestampValueExpr (#2066)
## Summary
This PR fixes the time filter expression generated when a source's timestamp value expression includes a Date-type column. In such cases, the bounds should be inclusive for the date-type columns. This extends the fix from #1915 to date-type columns.
### Screenshots or video
#### Before - Histogram is empty because of exclusive end bounds
<img width="1758" height="700" alt="Screenshot 2026-04-07 at 12 32 05 PM" src="https://github.com/user-attachments/assets/95898655-21f1-4380-9b23-5333fcea007f" />
#### After - Histogram is populated
<img width="1762" height="717" alt="Screenshot 2026-04-07 at 12 30 25 PM" src="https://github.com/user-attachments/assets/d159b131-1f4f-44f8-982e-b26ca68835ff" />
### How to test locally or on Vercel
The unit tests demonstrate the fix.
<details>
<summary>Alternatively, create a table with the following schema and data</summary>
```sql
CREATE TABLE default.otel_logs_date_pk
(
`Timestamp` DateTime64(9) CODEC(Delta(8), ZSTD(1)),
`TimestampTime` DateTime DEFAULT toDateTime(Timestamp),
`TimestampDate` Date DEFAULT toDate(Timestamp),
`TraceId` String CODEC(ZSTD(1)),
`SpanId` String CODEC(ZSTD(1)),
`TraceFlags` UInt8,
`SeverityText` LowCardinality(String) CODEC(ZSTD(1)),
`SeverityNumber` UInt8,
`ServiceName` LowCardinality(String) CODEC(ZSTD(1)),
`Body` String CODEC(ZSTD(1)),
`ResourceSchemaUrl` LowCardinality(String) CODEC(ZSTD(1)),
`ResourceAttributes` Map(LowCardinality(String), String) CODEC(ZSTD(1)),
`ScopeSchemaUrl` LowCardinality(String) CODEC(ZSTD(1)),
`ScopeName` String CODEC(ZSTD(1)),
`ScopeVersion` LowCardinality(String) CODEC(ZSTD(1)),
`ScopeAttributes` Map(LowCardinality(String), String) CODEC(ZSTD(1)),
`LogAttributes` Map(LowCardinality(String), String) CODEC(ZSTD(1)),
`__hdx_materialized_k8s.cluster.name` LowCardinality(String) MATERIALIZED ResourceAttributes['k8s.cluster.name'] CODEC(ZSTD(1)),
`__hdx_materialized_k8s.container.name` LowCardinality(String) MATERIALIZED ResourceAttributes['k8s.container.name'] CODEC(ZSTD(1)),
`__hdx_materialized_k8s.deployment.name` LowCardinality(String) MATERIALIZED ResourceAttributes['k8s.deployment.name'] CODEC(ZSTD(1)),
`__hdx_materialized_k8s.namespace.name` LowCardinality(String) MATERIALIZED ResourceAttributes['k8s.namespace.name'] CODEC(ZSTD(1)),
`__hdx_materialized_k8s.node.name` LowCardinality(String) MATERIALIZED ResourceAttributes['k8s.node.name'] CODEC(ZSTD(1)),
`__hdx_materialized_k8s.pod.name` LowCardinality(String) MATERIALIZED ResourceAttributes['k8s.pod.name'] CODEC(ZSTD(1)),
`__hdx_materialized_k8s.pod.uid` LowCardinality(String) MATERIALIZED ResourceAttributes['k8s.pod.uid'] CODEC(ZSTD(1)),
`__hdx_materialized_deployment.environment.name` LowCardinality(String) MATERIALIZED ResourceAttributes['deployment.environment.name'] CODEC(ZSTD(1)),
INDEX idx_trace_id TraceId TYPE bloom_filter(0.001) GRANULARITY 1,
INDEX idx_res_attr_key mapKeys(ResourceAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_res_attr_value mapValues(ResourceAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_scope_attr_key mapKeys(ScopeAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_scope_attr_value mapValues(ScopeAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_log_attr_key mapKeys(LogAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_log_attr_value mapValues(LogAttributes) TYPE bloom_filter(0.01) GRANULARITY 1,
INDEX idx_lower_body lower(Body) TYPE tokenbf_v1(32768, 3, 0) GRANULARITY 8
)
ENGINE = MergeTree
PARTITION BY toDate(TimestampTime)
PRIMARY KEY (TimestampDate, ServiceName, TimestampTime)
ORDER BY (TimestampDate, ServiceName, TimestampTime, Timestamp)
TTL TimestampTime + toIntervalDay(1)
SETTINGS index_granularity = 8192, ttl_only_drop_parts = 1;
insert into default.otel_logs_date_pk (Timestamp, SeverityText, Body) VALUES (now(), 'info', 'message');
insert into default.otel_logs_date_pk (Timestamp, SeverityText, Body) VALUES (now()-interval 1 minute, 'info', 'message');
```
Then create a source that points to that table with the timestampValueExpr `TimestampTime, TimestampDate`
</details>
### References
- Linear Issue: Closes HDX-3930
- Related PRs:1 parent ad71dc2 commit 24767c5
3 files changed
Lines changed: 99 additions & 7 deletions
File tree
- .changeset
- packages/common-utils/src
- __tests__
- core
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1293 | 1293 | | |
1294 | 1294 | | |
1295 | 1295 | | |
| 1296 | + | |
| 1297 | + | |
| 1298 | + | |
| 1299 | + | |
| 1300 | + | |
| 1301 | + | |
| 1302 | + | |
| 1303 | + | |
| 1304 | + | |
| 1305 | + | |
| 1306 | + | |
| 1307 | + | |
| 1308 | + | |
| 1309 | + | |
| 1310 | + | |
| 1311 | + | |
| 1312 | + | |
| 1313 | + | |
| 1314 | + | |
| 1315 | + | |
| 1316 | + | |
| 1317 | + | |
| 1318 | + | |
| 1319 | + | |
| 1320 | + | |
| 1321 | + | |
| 1322 | + | |
| 1323 | + | |
| 1324 | + | |
| 1325 | + | |
| 1326 | + | |
| 1327 | + | |
| 1328 | + | |
| 1329 | + | |
| 1330 | + | |
| 1331 | + | |
| 1332 | + | |
| 1333 | + | |
| 1334 | + | |
| 1335 | + | |
| 1336 | + | |
| 1337 | + | |
| 1338 | + | |
| 1339 | + | |
| 1340 | + | |
| 1341 | + | |
1296 | 1342 | | |
1297 | 1343 | | |
1298 | 1344 | | |
| |||
1338 | 1384 | | |
1339 | 1385 | | |
1340 | 1386 | | |
| 1387 | + | |
| 1388 | + | |
| 1389 | + | |
| 1390 | + | |
| 1391 | + | |
| 1392 | + | |
| 1393 | + | |
| 1394 | + | |
| 1395 | + | |
| 1396 | + | |
| 1397 | + | |
| 1398 | + | |
| 1399 | + | |
| 1400 | + | |
| 1401 | + | |
| 1402 | + | |
| 1403 | + | |
| 1404 | + | |
| 1405 | + | |
| 1406 | + | |
| 1407 | + | |
| 1408 | + | |
| 1409 | + | |
| 1410 | + | |
| 1411 | + | |
| 1412 | + | |
| 1413 | + | |
| 1414 | + | |
| 1415 | + | |
| 1416 | + | |
1341 | 1417 | | |
1342 | 1418 | | |
1343 | 1419 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
686 | 686 | | |
687 | 687 | | |
688 | 688 | | |
| 689 | + | |
| 690 | + | |
| 691 | + | |
689 | 692 | | |
690 | | - | |
| 693 | + | |
691 | 694 | | |
692 | 695 | | |
693 | 696 | | |
| |||
700 | 703 | | |
701 | 704 | | |
702 | 705 | | |
703 | | - | |
| 706 | + | |
| 707 | + | |
| 708 | + | |
| 709 | + | |
| 710 | + | |
| 711 | + | |
704 | 712 | | |
705 | 713 | | |
706 | 714 | | |
| |||
718 | 726 | | |
719 | 727 | | |
720 | 728 | | |
721 | | - | |
722 | | - | |
723 | | - | |
| 729 | + | |
| 730 | + | |
| 731 | + | |
| 732 | + | |
| 733 | + | |
| 734 | + | |
| 735 | + | |
724 | 736 | | |
725 | | - | |
726 | | - | |
| 737 | + | |
727 | 738 | | |
728 | 739 | | |
729 | 740 | | |
| |||
0 commit comments