Deleting Large Tables causes the system to exhaust connections pool or become unresponsive.

[krutoileshii]

Describe the bug

I have a very large table that was created through CDC. After executing DROP TABLE from trino, the whole polaris instance becomes unresponsive with logs like below:

Running polaris as a standalone (not docker).

2026-01-16 16:42:59,043 WARN  [org.apa.pol.ser.tas.FileCleanupTaskHandler] [,] [,,,] () file=s3://testdec16/allo_custom/dbo__cdc_test/data/00001-1768518467576-0d5953a4-ceed-49ed-a763-6ebb7b66a939-00001.parquet tableIdentifier=allo_custom.dbo__cdc_test baseFile=s3://testdec16/allo_custom/dbo__cdc_test/metadata/58be18c0-aa21-4b89-84dd-e1eb953f0444-m2.avro Exception caught deleting data file
2026-01-16 16:42:59,043 WARN  [org.apa.pol.ser.tas.FileCleanupTaskHandler] [,] [,,,] () file=s3://testdec16/allo_custom/dbo__cdc_test/data/00001-1768518467576-0d5953a4-ceed-49ed-a763-6ebb7b66a939-00001.parquet attempt=2 error=java.lang.RuntimeException: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool (SDK Attempt Count: 1) Error encountered attempting to delete file
2026-01-16 16:42:59,043 WARN  [org.apa.pol.ser.tas.FileCleanupTaskHandler] [,] [,,,] () file=s3://testdec16/allo_custom/dbo__cdc_test/data/00001-1768518467292-00defaac-2afc-4248-959c-f08f9294f161-00001.parquet tableIdentifier=allo_custom.dbo__cdc_test baseFile=s3://testdec16/allo_custom/dbo__cdc_test/metadata/58be18c0-aa21-4b89-84dd-e1eb953f0444-m2.avro Exception caught deleting data file
2026-01-16 16:42:59,043 WARN  [org.apa.pol.ser.tas.FileCleanupTaskHandler] [,] [,,,] () file=s3://testdec16/allo_custom/dbo__cdc_test/data/00001-1768518467292-00defaac-2afc-4248-959c-f08f9294f161-00001.parquet attempt=2 error=java.lang.RuntimeException: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool (SDK Attempt Count: 1) Error encountered attempting to delete file

After a while, it starts to delete them eventually, but during this process, you cannot access catalog, list tables, create table or query data.

Using

postgresql as a JDBC backend.

Here is my application config for using On-Prem S3.

polaris.realm-context.realms=polaris
quarkus.log.console.level=DEBUG
quarkus.log.category."software.amazon.awssdk.request".level=DEBUG
quarkus.log.category."software.amazon.awssdk.auth.signer".level=DEBUG
quarkus.log.category."software.amazon.awssdk.http".level=DEBUG
quarkus.log.category."io.smallrye.config".level=DEBUG
polaris.realm-context.header-name=Polaris-Realm
#polaris.realm-context.require-header=true
polaris.persistence.type=relational-jdbc
quarkus.datasource.db-kind=postgresql
quarkus.datasource.username=user_name
quarkus.datasource.password=password
quarkus.datasource.jdbc.url=jdbc:postgresql://postgresql:5432/polaris_test
polaris.authentication.token-broker.type=rsa-key-pair
polaris.authentication.token-broker.rsa-key-pair.public-key-file=/opt/polaris/public.key
polaris.authentication.token-broker.rsa-key-pair.private-key-file=/opt/polaris/private.key
polaris.authentication.token-broker.max-token-generation=PT1H
polaris.bootstrap.credentials=polaris,admin,creds
polaris.storage.aws.access-key=access_key
polaris.storage.aws.secret-key=secret_key
polaris.catalog.s3.no-sts=true
#polaris.catalog.s3.region=us-east-1
#polaris.features."SKIP_CREDENTIAL_SUBSCOPING_INDIRECTION": true
polaris.features."SUPPORTED_CATALOG_STORAGE_TYPES": ["S3"]
polaris.features."ALLOW_SETTING_S3_ENDPOINTS"=true
polaris.features."DROP_WITH_PURGE_ENABLED"=true
polaris.features."CLEANUP_ON_CATALOG_DROP"=true

To Reproduce

No response

Actual Behavior

No response

Expected Behavior

No response

Additional context

No response

System information

No response

[krutoileshii]

Also when deleting smaller files, getting errors like this:

2026-01-16 19:17:33,220 DEBUG [sof.ama.aws.htt.apa.int.con.IdleConnectionReaper] [,] [19644571311104ff186759821a23c2d3,,35498b5f90f77d70,true] (idle-connection-reaper) Shutting down reaper thread.
2026-01-16 19:17:33,217 DEBUG [sof.ama.aws.htt.apa.int.net.SdkSslSocket] [,] [19644571311104ff186759821a23c2d3,,35498b5f90f77d70,true] (executor-thread-2) closing objects./198:443
2026-01-16 19:17:35,171 WARN  [org.apa.pol.ser.tas.TaskExecutorImpl] [,] [,,,] (executor-thread-2) Failed to handle task entity id 9078206595966038194: java.util.concurrent.CompletionException: java.lang.IllegalArgumentException: Cannot read a delete manifest with a ManifestReader: GenericManifestFile{content=DELETES, path=s3://testdec16/allo_custom/finance__vendor_royalties/metadata/615c8533-b414-4ed1-bbfe-046dd88cbc60-m1.avro, length=7382, partition_spec_id=0, added_snapshot_id=6205443502928148023, added_data_files_count=1, added_rows_count=4095, existing_data_files_count=0, existing_rows_count=0, deleted_data_files_count=0, deleted_rows_count=0, partitions=[], key_metadata=null, sequence_number=1, min_sequence_number=1, first_row_id=null}
        at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)
        at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)
        at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1807)
        at io.smallrye.context.impl.wrappers.SlowContextualRunnable.run(SlowContextualRunnable.java:19)
        at org.jboss.threads.EnhancedViewExecutor$EnhancedViewExecutorRunnable.run(EnhancedViewExecutor.java:496)
        at io.quarkus.vertx.core.runtime.VertxCoreRecorder$15.runWith(VertxCoreRecorder.java:650)
        at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2651)
        at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2630)
        at org.jboss.threads.EnhancedQueueExecutor.runThreadBody(EnhancedQueueExecutor.java:1622)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1589)
        at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:11)
        at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:11)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.IllegalArgumentException: Cannot read a delete manifest with a ManifestReader: GenericManifestFile{content=DELETES, path=s3://testdec16/allo_custom/finance__vendor_royalties/metadata/615c8533-b414-4ed1-bbfe-046dd88cbc60-m1.avro, length=7382, partition_spec_id=0, added_snapshot_id=6205443502928148023, added_data_files_count=1, added_rows_count=4095, existing_data_files_count=0, existing_rows_count=0, deleted_data_files_count=0, deleted_rows_count=0, partitions=[], key_metadata=null, sequence_number=1, min_sequence_number=1, first_row_id=null}
        at org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkArgument(Preconditions.java:217)
        at org.apache.iceberg.ManifestFiles.read(ManifestFiles.java:127)
        at org.apache.iceberg.ManifestFiles.read(ManifestFiles.java:114)
        at org.apache.polaris.service.task.ManifestFileCleanupTaskHandler.cleanUpManifestFile(ManifestFileCleanupTaskHandler.java:83)
        at org.apache.polaris.service.task.ManifestFileCleanupTaskHandler.handleTask(ManifestFileCleanupTaskHandler.java:66)
        at org.apache.polaris.service.task.TaskExecutorImpl.handleTask(TaskExecutorImpl.java:172)
        at org.apache.polaris.service.task.TaskExecutorImpl.handleTaskWithTracing(TaskExecutorImpl.java:209)
        at org.apache.polaris.service.task.TaskExecutorImpl.lambda$tryHandleTask$0(TaskExecutorImpl.java:135)
        at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
        ... 11 more

2026-01-16 19:17:35,231 WARN  [org.apa.pol.ser.tas.TaskExecutorImpl] [,] [,,,] (executor-thread-3) Failed to handle task entity id 4837728316142291072: java.util.concurrent.CompletionException: java.lang.IllegalArgumentException: Cannot read a delete manifest with a ManifestReader: GenericManifestFile{content=DELETES, path=s3://testdec16/allo_custom/finance__vendor_royalties/metadata/70f25620-6b8d-441f-92a1-66773ed16cb3-m1.avro, length=7376, partition_spec_id=0, added_snapshot_id=7326900106277680047, added_data_files_count=1, added_rows_count=49, existing_data_files_count=0, existing_rows_count=0, deleted_data_files_count=0, deleted_rows_count=0, partitions=[], key_metadata=null, sequence_number=4, min_sequence_number=4, first_row_id=null}
        at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)
        at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)
        at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1807)
        at io.smallrye.context.impl.wrappers.SlowContextualRunnable.run(SlowContextualRunnable.java:19)
        at org.jboss.threads.EnhancedViewExecutor$EnhancedViewExecutorRunnable.run(EnhancedViewExecutor.java:496)
        at io.quarkus.vertx.core.runtime.VertxCoreRecorder$15.runWith(VertxCoreRecorder.java:650)
        at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2651)
        at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2630)
        at org.jboss.threads.EnhancedQueueExecutor.runThreadBody(EnhancedQueueExecutor.java:1622)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1589)
        at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:11)
        at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:11)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.IllegalArgumentException: Cannot read a delete manifest with a ManifestReader: GenericManifestFile{content=DELETES, path=s3://testdec16/allo_custom/finance__vendor_royalties/metadata/70f25620-6b8d-441f-92a1-66773ed16cb3-m1.avro, length=7376, partition_spec_id=0, added_snapshot_id=7326900106277680047, added_data_files_count=1, added_rows_count=49, existing_data_files_count=0, existing_rows_count=0, deleted_data_files_count=0, deleted_rows_count=0, partitions=[], key_metadata=null, sequence_number=4, min_sequence_number=4, first_row_id=null}  
        at org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkArgument(Preconditions.java:217)
        at org.apache.iceberg.ManifestFiles.read(ManifestFiles.java:127)
        at org.apache.iceberg.ManifestFiles.read(ManifestFiles.java:114)
        at org.apache.polaris.service.task.ManifestFileCleanupTaskHandler.cleanUpManifestFile(ManifestFileCleanupTaskHandler.java:83)
        at org.apache.polaris.service.task.ManifestFileCleanupTaskHandler.handleTask(ManifestFileCleanupTaskHandler.java:66)
        at org.apache.polaris.service.task.TaskExecutorImpl.handleTask(TaskExecutorImpl.java:172)
        at org.apache.polaris.service.task.TaskExecutorImpl.handleTaskWithTracing(TaskExecutorImpl.java:209)
        at org.apache.polaris.service.task.TaskExecutorImpl.lambda$tryHandleTask$0(TaskExecutorImpl.java:135)
        at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
        ... 11 more

2026-01-16 19:17:35,231 WARN  [org.apa.pol.ser.tas.TaskExecutorImpl] [,] [,,,] (executor-thread-2) Failed to handle task entity id 946636678825059231: java.util.concurrent.CompletionException: java.lang.IllegalArgumentException: Cannot read a delete manifest with a ManifestReader: GenericManifestFile{content=DELETES, path=s3://testdec16/allo_custom/finance__vendor_royalties/metadata/ca08dcd5-46c5-4ea4-9497-283c6b8e9841-m1.avro, length=7394, partition_spec_id=0, added_snapshot_id=5023809684943975341, added_data_files_count=1, added_rows_count=4095, existing_data_files_count=0, existing_rows_count=0, deleted_data_files_count=0, deleted_rows_count=0, partitions=[], key_metadata=null, sequence_number=2, min_sequence_number=2, first_row_id=null}
        at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)
        at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)
        at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1807)
        at io.smallrye.context.impl.wrappers.SlowContextualRunnable.run(SlowContextualRunnable.java:19)
        at org.jboss.threads.EnhancedViewExecutor$EnhancedViewExecutorRunnable.run(EnhancedViewExecutor.java:496)
        at io.quarkus.vertx.core.runtime.VertxCoreRecorder$15.runWith(VertxCoreRecorder.java:650)
        at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2651)
        at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2630)
        at org.jboss.threads.EnhancedQueueExecutor.runThreadBody(EnhancedQueueExecutor.java:1622)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1589)
        at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:11)
        at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:11)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.IllegalArgumentException: Cannot read a delete manifest with a ManifestReader: GenericManifestFile{content=DELETES, path=s3://testdec16/allo_custom/finance__vendor_royalties/metadata/ca08dcd5-46c5-4ea4-9497-283c6b8e9841-m1.avro, length=7394, partition_spec_id=0, added_snapshot_id=5023809684943975341, added_data_files_count=1, added_rows_count=4095, existing_data_files_count=0, existing_rows_count=0, deleted_data_files_count=0, deleted_rows_count=0, partitions=[], key_metadata=null, sequence_number=2, min_sequence_number=2, first_row_id=null}
        at org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkArgument(Preconditions.java:217)
        at org.apache.iceberg.ManifestFiles.read(ManifestFiles.java:127)
        at org.apache.iceberg.ManifestFiles.read(ManifestFiles.java:114)
        at org.apache.polaris.service.task.ManifestFileCleanupTaskHandler.cleanUpManifestFile(ManifestFileCleanupTaskHandler.java:83)
        at org.apache.polaris.service.task.ManifestFileCleanupTaskHandler.handleTask(ManifestFileCleanupTaskHandler.java:66)
        at org.apache.polaris.service.task.TaskExecutorImpl.handleTask(TaskExecutorImpl.java:172)
        at org.apache.polaris.service.task.TaskExecutorImpl.handleTaskWithTracing(TaskExecutorImpl.java:209)
        at org.apache.polaris.service.task.TaskExecutorImpl.lambda$tryHandleTask$0(TaskExecutorImpl.java:135)
        at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
        ... 11 more

[krutoileshii]

Could this be coming from here? If FileIO fails's wouldn't this still hold connection for manifest? Also, wouldn't it free up connections faster if we first got a manifest list and then did the FileIO to minimize pool usage. based on this it seems this will hold both sn.AllManifests map and then for the TaskUtils.exists.

private Stream<TaskEntity> getManifestTaskStream(
      TaskEntity cleanupTask,
      TableMetadata tableMetadata,
      FileIO fileIO,
      IcebergTableLikeEntity tableEntity,
      PolarisMetaStoreManager metaStoreManager,
      PolarisCallContext polarisCallContext) {
    // read the manifest list for each snapshot. dedupe the manifest files and schedule a
    // cleanupTask
    // for each manifest file and its data files to be deleted
    return tableMetadata.snapshots().stream()
        .flatMap(sn -> sn.allManifests(fileIO).stream())
        // distinct by manifest path, since multiple snapshots will contain the same
        // manifest
        .collect(Collectors.toMap(ManifestFile::path, Function.identity(), (mf1, mf2) -> mf1))
        .values()
        .stream()
        .filter(mf -> TaskUtils.exists(mf.path(), fileIO))
        .map(
            mf -> {
              // append a random uuid to the task name to avoid any potential conflict
              // when
              // storing the task entity. It's better to have duplicate tasks than to risk
              // not storing the rest of the task entities. If a duplicate deletion task
              // is
              // queued, it will check for the manifest file's existence and simply exit
              // if
              // the task has already been handled.
              String taskName = cleanupTask.getName() + "_" + mf.path() + "_" + UUID.randomUUID();


              LOGGER
                  .atDebug()
                  .addKeyValue("taskName", taskName)
                  .addKeyValue("tableIdentifier", tableEntity.getTableIdentifier())
                  .addKeyValue("metadataLocation", tableEntity.getMetadataLocation())
                  .addKeyValue("manifestFile", mf.path())
                  .log("Queueing task to delete manifest file");
              return new TaskEntity.Builder()
                  .setName(taskName)
                  .setId(metaStoreManager.generateNewEntityId(polarisCallContext).getId())
                  .setCreateTimestamp(clock.millis())
                  .withTaskType(AsyncTaskType.MANIFEST_FILE_CLEANUP)
                  .withData(
                      ManifestFileCleanupTaskHandler.ManifestCleanupTask.buildFrom(
                          tableEntity.getTableIdentifier(), mf))
                  .setId(metaStoreManager.generateNewEntityId(polarisCallContext).getId())
                  // copy the internal properties, which will have storage info
                  .setInternalProperties(cleanupTask.getInternalPropertiesAsMap())
                  .build();
            });
  }

[dimas-b]

@krutoileshii : Thanks for your interest in Polaris! I'm not sure many people have ready answers for this problem (like myself), so please do not consider lack of answers as discouragement 🙂 If you have specific fixes in mind, feel free to open PRs.

You may be interested in this discussion too: https://lists.apache.org/thread/0z8nb3w58zb9s617gsoyhzlnz53rt9zx

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment if you think it's still relevant.

[andrii29]

We encountered a similar problem. After dropping an Iceberg table with ~1000 partitions (~300 GB in size), Polaris became unresponsive and normal requests started to time out. Restarting Polaris pods helped. The table was deleted from Postgres, but the data wasn't cleared from S3 (MinIO).
Setup:
Polaris 1.5.0, 2 replicas, deployed via Helm chart

  AWS_REGION: us-west-2
  AWS_ACCESS_KEY_ID: polaris
  polaris.authentication.type: "mixed"
  polaris.features.DROP_WITH_PURGE_ENABLED: "true"
  polaris.features.CLEANUP_ON_CATALOG_DROP: "true"
  polaris.features.STORAGE_CREDENTIAL_DURATION_SECONDS: "36000"
  quarkus.otel.sdk.disabled: "true"
  QUARKUS_DATASOURCE_JDBC_MIN_SIZE: "10"
  QUARKUS_DATASOURCE_JDBC_MAX_SIZE: "85"
  QUARKUS_THREAD_POOL_CORE_THREADS: "10"

Sample error log line:

2026-07-07 14:46:57,007 ERROR [org.apa.pol.ser.exc.IcebergExceptionMapper] [5b47c38a-f4d4-4eac-b569-2c9389f96878_0000000000000154242,POLARIS] [,,,] (executor-thread-971) Unhandled exception returning INTERNAL_SERVER_ERROR: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool (SDK Attempt Count: 6)

metrics rate(http_server_requests_seconds_sum[$__rate_interval]) / rate(http_server_requests_seconds_count[$__rate_interval]) and http_server_requests_seconds_max

[andrii29]

I created a sample repository, trino-polaris-delete-connection-from-pool-timeout-issue, that reproduces the problem.

[shashihoushengqia]

I have also encountered this problem. May I ask if there is a solution?

[andrii29]

I have also encountered this problem. May I ask if there is a solution?

@shashihoushengqia I built an image from the main branch after 5004 was merged and achieved much better responsiveness from Polaris with the following options:

polaris.tasks.max-concurrent-tasks: "2"
polaris.storage.max-http-connections: "512"

However, Polaris somehow does not clean up the table metadata in the example repository.
I am waiting for a release so I can try it with real data.