Skip to content

Migration failure while updating #2325

Open

Description

@reckter

we just updated our hdx deployment from 2.24 to 2.27, and now the otel collector doesn't start, with these errors:

otel-collector-1  | 12:16:14 [seed] RETRY: Seed failed, retrying in 1s... (attempt 1/5): ERROR 00006_otel_logs_rollups.sql: failed to run SQL migration: failed to execute SQL query "CREATE MATERIALIZED VIEW IF NOT EXISTS default.otel_logs_attr_kv_rollup_15m_mv TO default.otel_logs_kv_rollup_15m\nAS WITH elements AS (\n    SELECT\n        'ResourceAttributes' AS ColumnIdentifier,\n        toStartOfFifteenMinutes(Timestamp) AS Timestamp,\n        replaceRegexpAll(entry.1, '\\\\[\\\\d+\\\\]', '[*]') AS Key,\n        CAST(entry.2 AS String) AS Value\n    FROM default.otel_logs\n    ARRAY JOIN ResourceAttributes AS entry\n    UNION ALL\n    SELECT\n        'LogAttributes' AS ColumnIdentifier,\n        toStartOfFifteenMinutes(Timestamp) AS Timestamp,\n        replaceRegexpAll(entry.1, '\\\\[\\\\d+\\\\]', '[*]') AS Key,\n        CAST(entry.2 AS String) AS Value\n    FROM default.otel_logs\n    ARRAY JOIN LogAttributes AS entry\n    UNION ALL\n    SELECT\n        'ScopeAttributes' AS ColumnIdentifier,\n        toStartOfFifteenMinutes(Timestamp) AS Timestamp,\n        replaceRegexpAll(entry.1, '\\\\[\\\\d+\\\\]', '[*]') AS Key,\n        CAST(entry.2 AS String) AS Value\n    FROM default.otel_logs\n    ARRAY JOIN ScopeAttributes AS entry\n    UNION ALL\n    SELECT\n        'NativeColumn' AS ColumnIdentifier,\n        toStartOfFifteenMinutes(Timestamp) AS Timestamp,\n        'SeverityText' as Key,\n        CAST(SeverityText AS String) as Value\n    FROM default.otel_logs\n    UNION ALL\n    SELECT\n        'NativeColumn' AS ColumnIdentifier,\n        toStartOfFifteenMinutes(Timestamp) AS Timestamp,\n        'ServiceName' as Key,\n        CAST(ServiceName AS String) as Value\n    FROM default.otel_logs\n    UNION ALL\n    SELECT\n        'NativeColumn' AS ColumnIdentifier,\n        toStartOfFifteenMinutes(Timestamp) AS Timestamp,\n        'ScopeName' as Key,\n        CAST(ScopeName AS String) as Value\n    FROM default.otel_logs\n    UNION ALL\n    SELECT\n        'NativeColumn' AS ColumnIdentifier,\n        toStartOfFifteenMinutes(Timestamp) AS Timestamp,\n        'ScopeVersion' as Key,\n        CAST(ScopeVersion AS String) as Value\n    FROM default.otel_logs\n    UNION ALL\n    SELECT\n        'NativeColumn' AS ColumnIdentifier,\n        toStartOfFifteenMinutes(Timestamp) AS Timestamp,\n        'ResourceSchemaUrl' as Key,\n        CAST(ResourceSchemaUrl AS String) as Value\n    FROM default.otel_logs\n    UNION ALL\n    SELECT\n        'NativeColumn' AS ColumnIdentifier,\n        toStartOfFifteenMinutes(Timestamp) AS Timestamp,\n        'ScopeSchemaUrl' as Key,\n        CAST(ScopeSchemaUrl AS String) as Value\n    FROM default.otel_logs\n    UNION ALL\n    SELECT\n        'NativeColumn' AS ColumnIdentifier,\n        toStartOfFifteenMinutes(Timestamp) AS Timestamp,\n        '__hdx_materialized_k8s.cluster.name' as Key,\n        CAST(`__hdx_materialized_k8s.cluster.name` AS String) as Value\n    FROM default.otel_logs\n    UNION ALL\n    SELECT\n        'NativeColumn' AS ColumnIdentifier,\n        toStartOfFifteenMinutes(Timestamp) AS Timestamp,\n        '__hdx_materialized_k8s.container.name' as Key,\n        CAST(`__hdx_materialized_k8s.container.name` AS String) as Value\n    FROM default.otel_logs\n    UNION ALL\n    SELECT\n        'NativeColumn' AS ColumnIdentifier,\n        toStartOfFifteenMinutes(Timestamp) AS Timestamp,\n        '__hdx_materialized_k8s.deployment.name' as Key,\n        CAST(`__hdx_materialized_k8s.deployment.name` AS String) as Value\n    FROM default.otel_logs\n    UNION ALL\n    SELECT\n        'NativeColumn' AS ColumnIdentifier,\n        toStartOfFifteenMinutes(Timestamp) AS Timestamp,\n        '__hdx_materialized_k8s.namespace.name' as Key,\n        CAST(`__hdx_materialized_k8s.namespace.name` AS String) as Value\n    FROM default.otel_logs\n    UNION ALL\n    SELECT\n        'NativeColumn' AS ColumnIdentifier,\n        toStartOfFifteenMinutes(Timestamp) AS Timestamp,\n        '__hdx_materialized_k8s.node.name' as Key,\n        CAST(`__hdx_materialized_k8s.node.name` AS String) as Value\n    FROM default.otel_logs\n    UNION ALL\n    SELECT\n        'NativeColumn' AS ColumnIdentifier,\n        toStartOfFifteenMinutes(Timestamp) AS Timestamp,\n        '__hdx_materialized_k8s.pod.name' as Key,\n        CAST(`__hdx_materialized_k8s.pod.name` AS String) as Value\n    FROM default.otel_logs\n    UNION ALL\n    SELECT\n        'NativeColumn' AS ColumnIdentifier,\n        toStartOfFifteenMinutes(Timestamp) AS Timestamp,\n        '__hdx_materialized_k8s.pod.uid' as Key,\n        CAST(`__hdx_materialized_k8s.pod.uid` AS String) as Value\n    FROM default.otel_logs\n    UNION ALL\n    SELECT\n        'NativeColumn' AS ColumnIdentifier,\n        toStartOfFifteenMinutes(Timestamp) AS Timestamp,\n        '__hdx_materialized_deployment.environment.name' as Key,\n        CAST(`__hdx_materialized_deployment.environment.name` AS String) as Value\n    FROM default.otel_logs\n)\nSELECT Timestamp, ColumnIdentifier, Key, Value, count() AS count FROM elements\nGROUP BY Timestamp, ColumnIdentifier, Key, Value;": code: 47, message: Unknown expression or function identifier `__hdx_materialized_k8s.cluster.name` in scope SELECT 'NativeColumn' AS ColumnIdentifier, toStartOfFifteenMinutes(Timestamp) AS Timestamp, '__hdx_materialized_k8s.cluster.name' AS Key, CAST(`__hdx_materialized_k8s.cluster.name`, 'String') AS Value FROM default.otel_logs

I could downgrade again to 2.24 for now.
I couldn't find any upgrade migration todos, so what do we need to do in order to update?

Activity

  1. czephyr commented on Jun 1, 2026

    @czephyr

    Probably related to #2297

  2. reckter commented on Jul 9, 2026

    @reckter
    Author

    I just tried to update to 2.29 (from 2.24, same install as above), and got the same problem.

    @czephyr in the ticket @pulpdrew mentioned that these migrations should not run on existing deployments.
    But apparently do in our case? (I might not unterstand it correctly though?)

  3. czephyr commented on Jul 9, 2026

    @czephyr

    @reckter wild guess, but possibly whats happening for you is that the otel-collector clickhouse exporter has create_schema: true and its trying to push the new schema over your old one?

  4. pulpdrew commented on Jul 9, 2026

    @pulpdrew
    Contributor

    @knudtty this looks like one of the new metadata rollup views - do you know what might be happening here?

  5. knudtty commented on Jul 9, 2026

    @knudtty
    Contributor

    Interesting, I'll try and reproduce this. If you add the env var HYPERDX_OTEL_EXPORTER_CREATE_LEGACY_SCHEMA=true you can probably avoid this error, but that's pretty hacky IMO.

  6. knudtty commented on Jul 10, 2026

    @knudtty
    Contributor

    I tried to reproduce by running from 2.24.0, then upgrading to 2.29.0. However, the migrations looked like they finished correctly.

    14:23:07 [seed]   - 00001_create_database.sql
    14:23:07 [seed]   - 00002_otel_logs.sql
    14:23:07 [seed]   - 00003_otel_metrics.sql
    14:23:07 [seed]   - 00004_hyperdx_sessions.sql
    14:23:07 [seed]   - 00005_otel_traces.sql
    14:23:07 [seed]   - 00006_otel_logs_rollups.sql
    14:23:07 [seed]   - 00007_otel_traces_rollups.sql
    14:23:07 [seed] OK   00001_create_database.sql (1.81ms)
    14:23:07 [seed] OK   00002_otel_logs.sql (4.43ms)
    14:23:07 [seed] OK   00003_otel_metrics.sql (8.01ms)
    14:23:07 [seed] OK   00004_hyperdx_sessions.sql (1.02ms)
    14:23:07 [seed] OK   00005_otel_traces.sql (2.08ms)
    14:23:07 [seed] OK   00006_otel_logs_rollups.sql (29.74ms)
    14:23:07 [seed] OK   00007_otel_traces_rollups.sql (24.09ms)
    

    Could you provide some more info? Can you paste your logs and traces schemas? Command to get them is SHOW CREATE otel_logs and SHOW CREATE otel_traces

  7. mhaley-tignis commented on Sep 17, 2026

    @mhaley-tignis

    Ran into this and proposed a solution here #3141.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions