List view

Is it possible to support PyArrow backed UDFs in Comet natively?

enhancementNew feature or requestNew feature or request

Status: Open.

#957 In apache/datafusion-comet;

· SemyonSinchenko opened on Sep 21, 2024

Tighten CometUDF API with input/return type validation at registration

area:expressionsExpression evaluationExpression evaluation

priority:mediumFunctional bugs, performance regressions, broken featuresFunctional bugs, performance regressions, broken features

Status: Open.

#4173 In apache/datafusion-comet;

· andygrove opened on May 1, 2026

Register CometArrowAllocator as a Spark MemoryConsumer for JVM-UDF dispatch

area:expressionsExpression evaluationExpression evaluation

area:ffiArrow FFI / JNI boundaryArrow FFI / JNI boundary

priority:mediumFunctional bugs, performance regressions, broken featuresFunctional bugs, performance regressions, broken features

Status: Open.

#4174 In apache/datafusion-comet;

· andygrove opened on May 1, 2026

Make JVM-scalar-UDF dispatch responsive to task cancellation

area:expressionsExpression evaluationExpression evaluation

priority:mediumFunctional bugs, performance regressions, broken featuresFunctional bugs, performance regressions, broken features

Status: Open.

#4175 In apache/datafusion-comet;

· andygrove opened on May 1, 2026

Add metrics and logging to JVM-scalar-UDF dispatch path

area:expressionsExpression evaluationExpression evaluation

priority:lowMinor issues, test failures, tooling, cosmeticMinor issues, test failures, tooling, cosmetic

Status: Open.

#4176 In apache/datafusion-comet;

· andygrove opened on May 1, 2026

Add support for scalar UDFs that operate on Arrow data

area:expressionsExpression evaluationExpression evaluation

enhancementNew feature or requestNew feature or request

priority:mediumFunctional bugs, performance regressions, broken featuresFunctional bugs, performance regressions, broken features

Status: Open.

#4177 In apache/datafusion-comet;

· andygrove opened on May 1, 2026

Support higher-order array functions via JVM UDF bridge

requires-triage

Status: Open.

#4224 In apache/datafusion-comet;

· andygrove opened on May 5, 2026

Eliminate remaining row<->Arrow round-trip in CometPythonMapInArrowExec

enhancementNew feature or requestNew feature or request

requires-triage

Status: Open.

#4240 In apache/datafusion-comet;

· andygrove opened on May 6, 2026

Disable Comet if shuffle manager not registered

enhancementNew feature or requestNew feature or request

performance

Status: Open.

#3900 In apache/datafusion-comet;

· andygrove opened on Apr 3, 2026

[Research] Use custom cost model when deciding between SMJ and SHJ

enhancementNew feature or requestNew feature or request

performance

Status: Open.

#1011 In apache/datafusion-comet;

· andygrove opened on Oct 10, 2024

Add support for RegExpExtract and RegExpExtractAll

enhancementNew feature or requestNew feature or request

expressionSpark expression relatedSpark expression related

good first issueGood for newcomersGood for newcomers

Status: Open.

#2708 In apache/datafusion-comet;

· andygrove opened on Nov 6, 2025

Improve performance of aggregate expressions by implementing group accumulators

enhancementNew feature or requestNew feature or request

performance

Status: Open.

#1627 In apache/datafusion-comet;

· andygrove opened on Apr 8, 2025

feat: Add direct native shuffle execution optimization

area:ffiArrow FFI / JNI boundaryArrow FFI / JNI boundary

performance

Status: Draft (not ready).

apache/datafusion-cometnumber 3230

#3230 In apache/datafusion-comet;

· andygrove opened on Jan 21, 2026

[EPIC] Improve performance of TPC-DS queries

enhancementNew feature or requestNew feature or request

performance

Status: Open.

#858 In apache/datafusion-comet;

· andygrove opened on Aug 21, 2024

Re-enable COUNT for mixed Spark partial / Comet final aggregate execution

requires-triage

Status: Open.

#4242 In apache/datafusion-comet;

· andygrove opened on May 6, 2026

Various issues with Comet support for HashAggregateExec

enhancementNew feature or requestNew feature or request

Status: Open.

#1267 In apache/datafusion-comet;

· andygrove opened on Jan 10, 2025

Improve fallback reporting for aggregates

enhancementNew feature or requestNew feature or request

Status: Open.

#2787 In apache/datafusion-comet;

· andygrove opened on Nov 14, 2025

Bloom filter intermediate aggregate buffers are not compatible between Spark and Comet

area:aggregationHash aggregates, aggregate expressionsHash aggregates, aggregate expressions

bugSomething isn't workingSomething isn't working

crashNative engine crash/panic/segfaultNative engine crash/panic/segfault

priority:criticalData corruption, silent wrong results, security issuesData corruption, silent wrong results, security issues

Status: Open.

#2889 In apache/datafusion-comet;

· andygrove opened on Dec 12, 2025

[EPIC] Improve planning of hash aggregates

area:aggregationHash aggregates, aggregate expressionsHash aggregates, aggregate expressions

bugSomething isn't workingSomething isn't working

priority:lowMinor issues, test failures, tooling, cosmeticMinor issues, test failures, tooling, cosmetic

Status: Open.

#2892 In apache/datafusion-comet;

· andygrove opened on Dec 12, 2025

Comet falls back to Spark for final hash aggregate in some cases when it could be supported

enhancementNew feature or requestNew feature or request

Status: Open.

#2894 In apache/datafusion-comet;

· andygrove opened on Dec 13, 2025

Improve performance of first/last aggregates

enhancementNew feature or requestNew feature or request

performance

Status: Open.

#3022 In apache/datafusion-comet;

· andygrove opened on Jan 2, 2026