`array_union` correctness issue

[comphead]

Describe the bug

Error executing SQL 'SELECT array_union(a, b) FROM test_array_union'

== Results ==
!== Correct Answer - 5 ==               == Spark Answer - 5 ==
 struct<array_union(a, b):array<int>>   struct<array_union(a, b):array<int>>
 [ArrayBuffer(1)]                       [ArrayBuffer(1)]
 [ArrayBuffer(1, 2)]                    [ArrayBuffer(1, 2)]
 [ArrayBuffer(1, 2, 3, 4, 5)]           [ArrayBuffer(1, 2, 3, 4, 5)]
![ArrayBuffer(1, null, 2)]              [ArrayBuffer(null, 1, 2)]
 [null]                                 [null]

Steps to reproduce

No response

Expected behavior

No response

Additional context

No response

[n0r0shi]

I think this is fixed in DataFusion 53.0.0 by PR apache/datafusion#20243, which replaced .sorted().dedup() with a HashSet for dedup tracking + a Vec for collecting results, preserving the natural left-then-right encounter order.

Will this be resolved automatically once Comet upgrades to DataFusion 53 (#3620)?

[comphead]

Will this be resolved automatically once Comet upgrades to DataFusion 53 (#3620)?

it might be, unfortunately we cannot check df-53 right now because of some pending dependencies, lets try it once again after df53 dependencies resolved

[andygrove]

This expression (ArrayUnion) is now marked as Incompatible in Comet, which means it falls back to Spark by default and only runs natively when spark.comet.expr.allowIncompatible=true is explicitly set. Since default behavior is now correct (via Spark fallback), this is no longer a bug. Relabeling as an enhancement to track implementing compatible native behavior in the future.