Repository navigation
Open
Description
we just updated our hdx deployment from 2.24 to 2.27, and now the otel collector doesn't start, with these errors:
otel-collector-1 | 12:16:14 [seed] RETRY: Seed failed, retrying in 1s... (attempt 1/5): ERROR 00006_otel_logs_rollups.sql: failed to run SQL migration: failed to execute SQL query "CREATE MATERIALIZED VIEW IF NOT EXISTS default.otel_logs_attr_kv_rollup_15m_mv TO default.otel_logs_kv_rollup_15m\nAS WITH elements AS (\n SELECT\n 'ResourceAttributes' AS ColumnIdentifier,\n toStartOfFifteenMinutes(Timestamp) AS Timestamp,\n replaceRegexpAll(entry.1, '\\\\[\\\\d+\\\\]', '[*]') AS Key,\n CAST(entry.2 AS String) AS Value\n FROM default.otel_logs\n ARRAY JOIN ResourceAttributes AS entry\n UNION ALL\n SELECT\n 'LogAttributes' AS ColumnIdentifier,\n toStartOfFifteenMinutes(Timestamp) AS Timestamp,\n replaceRegexpAll(entry.1, '\\\\[\\\\d+\\\\]', '[*]') AS Key,\n CAST(entry.2 AS String) AS Value\n FROM default.otel_logs\n ARRAY JOIN LogAttributes AS entry\n UNION ALL\n SELECT\n 'ScopeAttributes' AS ColumnIdentifier,\n toStartOfFifteenMinutes(Timestamp) AS Timestamp,\n replaceRegexpAll(entry.1, '\\\\[\\\\d+\\\\]', '[*]') AS Key,\n CAST(entry.2 AS String) AS Value\n FROM default.otel_logs\n ARRAY JOIN ScopeAttributes AS entry\n UNION ALL\n SELECT\n 'NativeColumn' AS ColumnIdentifier,\n toStartOfFifteenMinutes(Timestamp) AS Timestamp,\n 'SeverityText' as Key,\n CAST(SeverityText AS String) as Value\n FROM default.otel_logs\n UNION ALL\n SELECT\n 'NativeColumn' AS ColumnIdentifier,\n toStartOfFifteenMinutes(Timestamp) AS Timestamp,\n 'ServiceName' as Key,\n CAST(ServiceName AS String) as Value\n FROM default.otel_logs\n UNION ALL\n SELECT\n 'NativeColumn' AS ColumnIdentifier,\n toStartOfFifteenMinutes(Timestamp) AS Timestamp,\n 'ScopeName' as Key,\n CAST(ScopeName AS String) as Value\n FROM default.otel_logs\n UNION ALL\n SELECT\n 'NativeColumn' AS ColumnIdentifier,\n toStartOfFifteenMinutes(Timestamp) AS Timestamp,\n 'ScopeVersion' as Key,\n CAST(ScopeVersion AS String) as Value\n FROM default.otel_logs\n UNION ALL\n SELECT\n 'NativeColumn' AS ColumnIdentifier,\n toStartOfFifteenMinutes(Timestamp) AS Timestamp,\n 'ResourceSchemaUrl' as Key,\n CAST(ResourceSchemaUrl AS String) as Value\n FROM default.otel_logs\n UNION ALL\n SELECT\n 'NativeColumn' AS ColumnIdentifier,\n toStartOfFifteenMinutes(Timestamp) AS Timestamp,\n 'ScopeSchemaUrl' as Key,\n CAST(ScopeSchemaUrl AS String) as Value\n FROM default.otel_logs\n UNION ALL\n SELECT\n 'NativeColumn' AS ColumnIdentifier,\n toStartOfFifteenMinutes(Timestamp) AS Timestamp,\n '__hdx_materialized_k8s.cluster.name' as Key,\n CAST(`__hdx_materialized_k8s.cluster.name` AS String) as Value\n FROM default.otel_logs\n UNION ALL\n SELECT\n 'NativeColumn' AS ColumnIdentifier,\n toStartOfFifteenMinutes(Timestamp) AS Timestamp,\n '__hdx_materialized_k8s.container.name' as Key,\n CAST(`__hdx_materialized_k8s.container.name` AS String) as Value\n FROM default.otel_logs\n UNION ALL\n SELECT\n 'NativeColumn' AS ColumnIdentifier,\n toStartOfFifteenMinutes(Timestamp) AS Timestamp,\n '__hdx_materialized_k8s.deployment.name' as Key,\n CAST(`__hdx_materialized_k8s.deployment.name` AS String) as Value\n FROM default.otel_logs\n UNION ALL\n SELECT\n 'NativeColumn' AS ColumnIdentifier,\n toStartOfFifteenMinutes(Timestamp) AS Timestamp,\n '__hdx_materialized_k8s.namespace.name' as Key,\n CAST(`__hdx_materialized_k8s.namespace.name` AS String) as Value\n FROM default.otel_logs\n UNION ALL\n SELECT\n 'NativeColumn' AS ColumnIdentifier,\n toStartOfFifteenMinutes(Timestamp) AS Timestamp,\n '__hdx_materialized_k8s.node.name' as Key,\n CAST(`__hdx_materialized_k8s.node.name` AS String) as Value\n FROM default.otel_logs\n UNION ALL\n SELECT\n 'NativeColumn' AS ColumnIdentifier,\n toStartOfFifteenMinutes(Timestamp) AS Timestamp,\n '__hdx_materialized_k8s.pod.name' as Key,\n CAST(`__hdx_materialized_k8s.pod.name` AS String) as Value\n FROM default.otel_logs\n UNION ALL\n SELECT\n 'NativeColumn' AS ColumnIdentifier,\n toStartOfFifteenMinutes(Timestamp) AS Timestamp,\n '__hdx_materialized_k8s.pod.uid' as Key,\n CAST(`__hdx_materialized_k8s.pod.uid` AS String) as Value\n FROM default.otel_logs\n UNION ALL\n SELECT\n 'NativeColumn' AS ColumnIdentifier,\n toStartOfFifteenMinutes(Timestamp) AS Timestamp,\n '__hdx_materialized_deployment.environment.name' as Key,\n CAST(`__hdx_materialized_deployment.environment.name` AS String) as Value\n FROM default.otel_logs\n)\nSELECT Timestamp, ColumnIdentifier, Key, Value, count() AS count FROM elements\nGROUP BY Timestamp, ColumnIdentifier, Key, Value;": code: 47, message: Unknown expression or function identifier `__hdx_materialized_k8s.cluster.name` in scope SELECT 'NativeColumn' AS ColumnIdentifier, toStartOfFifteenMinutes(Timestamp) AS Timestamp, '__hdx_materialized_k8s.cluster.name' AS Key, CAST(`__hdx_materialized_k8s.cluster.name`, 'String') AS Value FROM default.otel_logs
I could downgrade again to 2.24 for now.
I couldn't find any upgrade migration todos, so what do we need to do in order to update?
Reacted by LocalXpose, Toh Yong Cheng, Alfredo Funicello, Hakizu and Mark Haley
Metadata
Metadata
Assignees
Labels
No labels
Activity
czephyr commentedon Jun 1, 2026
Probably related to #2297
reckter commentedon Jul 9, 2026
I just tried to update to 2.29 (from 2.24, same install as above), and got the same problem.
@czephyr in the ticket @pulpdrew mentioned that these migrations should not run on existing deployments.
But apparently do in our case? (I might not unterstand it correctly though?)
czephyr commentedon Jul 9, 2026
@reckter wild guess, but possibly whats happening for you is that the otel-collector clickhouse exporter has
create_schema: trueand its trying to push the new schema over your old one?pulpdrew commentedon Jul 9, 2026
@knudtty this looks like one of the new metadata rollup views - do you know what might be happening here?
knudtty commentedon Jul 9, 2026
Interesting, I'll try and reproduce this. If you add the env var
HYPERDX_OTEL_EXPORTER_CREATE_LEGACY_SCHEMA=trueyou can probably avoid this error, but that's pretty hacky IMO.knudtty commentedon Jul 10, 2026
I tried to reproduce by running from 2.24.0, then upgrading to 2.29.0. However, the migrations looked like they finished correctly.
Could you provide some more info? Can you paste your logs and traces schemas? Command to get them is
SHOW CREATE otel_logsandSHOW CREATE otel_tracesmhaley-tignis commentedon Sep 17, 2026
Ran into this and proposed a solution here #3141.