Dremio Query task errors out with `CONNECTION : SSL negotiation failed`

[anna-geller]

Feature description

To reproduce, run:

docker run --pull=always --rm -it -p 28080:8080 --user=root -v /var/run/docker.sock:/var/run/docker.sock -e SECRET_DREMIO_TOKEN="SmFORmFrY1RUOUdwTlh4SkFjNDJSbXBFYUVCSS9NTi9NZVNzUGFyTlVUYTV0QUxZNytoSTVRcGI4ckVvZXc9PQ==" kestra/kestra:develop-full server local

and run this flow:

id: dremio_sql_python
namespace: dremio

variables:
  user: data.engineering.blog@gmail.com
  project_id: ead79cc0-9e93-4d50-b364-77639a56d4a6

tasks:
  - id: wdir
    type: io.kestra.core.tasks.flows.WorkingDirectory
    tasks:
    - id: cloneRepository
      type: io.kestra.plugin.git.Clone
      url: https://github.com/dbt-labs/jaffle_shop
      branch: main

    - id: dbt
      type: io.kestra.plugin.dbt.cli.DbtCLI
      docker:
        image: ghcr.io/kestra-io/dbt-dremio:latest
      profiles: |
        jaffle_shop:
          outputs:
            dev:
              type: dremio
              threads: 16
              cloud_host: api.dremio.cloud
              cloud_project_id: "{{ vars.project_id }}"
              user: "{{ vars.user }}"
              pat: "{{ secret('DREMIO_TOKEN') }}"
              use_ssl: true
          target: dev
      commands:
        - dbt build

  - id: query
    type: io.kestra.plugin.jdbc.dremio.Query
    url: "jdbc:dremio:direct=sql.dremio.cloud:443;ssl=true;PROJECT_ID={{vars.project_id}};"
    username: $token
    password: "{{ secret('DREMIO_TOKEN') }}"
    sql: SELECT * FROM "@{{ vars.user }}"."stg_customers";
    store: true

  - id: to_csv
    type: io.kestra.plugin.serdes.csv.CsvWriter
    from: "{{ outputs.query.uri }}"

  - id: python
    type: io.kestra.plugin.scripts.python.Script
    warningOnStdErr: false
    docker:
      image: ghcr.io/kestra-io/polars:latest
    inputFiles:
      data.csv: "{{ outputs.to_csv.uri }}"
    script: |
      import polars as pl
      df = pl.read_csv("data.csv")

      print(df.glimpse())

full stack-trace:

java.sql.SQLException: Failure in connecting to Dremio: cdjd.com.dremio.exec.rpc.ConnectionFailedException: CONNECTION : SSL negotiation failed
	at com.dremio.jdbc.impl.DremioExceptionMapper.map(DremioExceptionMapper.java:81)
	at com.dremio.jdbc.impl.DremioConnectionImpl.<init>(DremioConnectionImpl.java:115)
	at com.dremio.jdbc.impl.DremioJdbc41Factory.newConnection(DremioJdbc41Factory.java:72)
	at com.dremio.jdbc.impl.DremioFactory.newConnection(DremioFactory.java:67)
	at cdjd.org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)
	at com.dremio.jdbc.Driver.connect(Driver.java:70)
	at java.sql/java.sql.DriverManager.getConnection(Unknown Source)
	at java.sql/java.sql.DriverManager.getConnection(Unknown Source)
	at io.kestra.plugin.jdbc.JdbcConnectionInterface.connection(JdbcConnectionInterface.java:63)
	at io.kestra.plugin.jdbc.AbstractJdbcQuery.run(AbstractJdbcQuery.java:77)
	at io.kestra.plugin.jdbc.dremio.Query.run(Query.java:55)
	at io.kestra.plugin.jdbc.dremio.Query.run(Query.java:20)
	at io.kestra.core.runners.Worker$WorkerThread.run(Worker.java:727)
Caused by: cdjd.com.dremio.exec.rpc.ConnectionFailedException: CONNECTION : SSL negotiation failed
	at cdjd.com.dremio.exec.rpc.ConnectionFailedException.mapException(ConnectionFailedException.java:31)
	at cdjd.com.dremio.exec.client.DremioClient$FutureHandler.connectionFailed(DremioClient.java:830)
	at cdjd.com.dremio.sabot.rpc.user.QueryResultHandler$ChannelClosedHandler.connectionFailed(QueryResultHandler.java:387)
	at cdjd.com.dremio.exec.rpc.BasicClient$ConnectionMultiListener$ConnectionEstablishmentListener.lambda$addNegotiator$0(BasicClient.java:341)
	at cdjd.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577)
	at cdjd.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:551)
	at cdjd.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490)
	at cdjd.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615)
	at cdjd.io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:608)
	at cdjd.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117)
	at cdjd.io.netty.handler.ssl.SslHandler.setHandshakeFailure(SslHandler.java:1804)
	at cdjd.io.netty.handler.ssl.SslHandler.setHandshakeFailure(SslHandler.java:1773)
	at cdjd.io.netty.handler.ssl.SslHandler.handshake(SslHandler.java:1996)
	at cdjd.io.netty.handler.ssl.SslHandler.startHandshakeProcessing(SslHandler.java:1914)
	at cdjd.io.netty.handler.ssl.SslHandler.handlerAdded(SslHandler.java:1903)
	at cdjd.io.netty.channel.AbstractChannelHandlerContext.callHandlerAdded(AbstractChannelHandlerContext.java:971)
	at cdjd.io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:609)
	at cdjd.io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:181)
	at cdjd.io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:152)
	at cdjd.com.dremio.exec.rpc.BasicClient$ConnectionMultiListener$ConnectionEstablishmentListener.addNegotiator(BasicClient.java:350)
	at cdjd.com.dremio.exec.rpc.BasicClient$ConnectionMultiListener$ConnectionEstablishmentListener.operationComplete(BasicClient.java:291)
	at cdjd.com.dremio.exec.rpc.BasicClient$ConnectionMultiListener$ConnectionEstablishmentListener.operationComplete(BasicClient.java:272)
	at cdjd.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577)
	at cdjd.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:570)
	at cdjd.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:549)
	at cdjd.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490)
	at cdjd.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615)
	at cdjd.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:604)
	at cdjd.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
	at cdjd.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84)
	at cdjd.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:300)
	at cdjd.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:335)
	at cdjd.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
	at cdjd.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
	at cdjd.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
	at cdjd.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
	at cdjd.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at cdjd.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: cdjd.com.dremio.exec.rpc.RpcException: CONNECTION : SSL negotiation failed
	at cdjd.com.dremio.exec.client.DremioClient$FutureHandler.connectionFailed(DremioClient.java:826)
	... 37 common frames omitted
Caused by: cdjd.com.dremio.exec.rpc.RpcException: SSL negotiation failed
	at cdjd.com.dremio.exec.rpc.BasicClient$ConnectionMultiListener$ConnectionEstablishmentListener.lambda$addNegotiator$0(BasicClient.java:342)
	... 35 common frames omitted
Caused by: java.lang.UnsupportedOperationException: Allocator doesn't support heap-based memory.
	at cdjd.org.apache.arrow.memory.ArrowByteBufAllocator.fail(ArrowByteBufAllocator.java:158)
	at cdjd.org.apache.arrow.memory.ArrowByteBufAllocator.heapBuffer(ArrowByteBufAllocator.java:129)
	at cdjd.io.netty.handler.ssl.SslHandler$SslEngineType$3.allocateWrapBuffer(SslHandler.java:311)
	at cdjd.io.netty.handler.ssl.SslHandler.allocateOutNetBuf(SslHandler.java:2162)
	at cdjd.io.netty.handler.ssl.SslHandler.wrapNonAppData(SslHandler.java:941)
	at cdjd.io.netty.handler.ssl.SslHandler.handshake(SslHandler.java:1994)
	... 26 common frames omitted

[anna-geller]

maybe we need to switch to PAT?

  - id: query
    type: io.kestra.plugin.jdbc.dremio.Query
    url: "jdbc:dremio:direct=sql.dremio.cloud:443;token_type=personal_access_token;ssl=true;PROJECT_ID={{vars.project_id}};"

https://docs.dremio.com/cloud/sonar/client-apps/drivers/jdbc/

[Skraye]

Revert driver update temporary so it works, will need further investigation why latest driver does not work with the cloud :
78f3857