Unifying Operator Handling with the Scalar Function Framework

[Acfboy]

Is your feature request related to a problem or challenge?

While working on #11250, I noticed that the current handling of operators is quite scattered and complex. This is the root cause of why operator diagnostic messages are less user-friendly than those of scalar functions, and it may creates maintenance hurdles.

For example:

• For SELECT 2.0 << 3.5;, constant folding occurs during the optimization phase, the type mismatch is caught in evaluate_with_resolved_args.
• But for SELECT 1 + 'a';, the error is caught during projection_schema in the LogicalPlan phase.:
• And for SELECT a << b FROM ...;, the error not be surfaced until the physical execution phase in evaluate_expressions_to_arrays_with_metrics.

Describe the solution you'd like

I propose that we gradually refactor operator handling by bringing it under the same framework as scalar functions.

In my initial thinking, the roadmap is like this:

1. Start by defining signatures for simple binary operators. This allows us to reuse the existing function-based type coercion logic during LogicalPlan generation.
2. Implement rewrite rules to gradually transform these operators into function-based calls.
3. Extend this to more unary and binary operations.
4. Finally, address operators like LIKE, which have unique syntax or optimization paths compared to standard binary operators.

Describe alternatives you've considered

No response

Additional context

What do the maintainers think about this direction? I would love to hear your thoughts on whether this unification aligns with the long-term vision of DataFusion, or suggestions on task decomposition and how to best phase this refactor.

[2010YOUY01]

The high level idea I think is perfect, unifying them can make a lot of implementation simpler.

However I'm not sure how hard the implementation would be, I suggest we try to move one operator first to see how things go, specifically I think it is implementing operator logic through ScalarUDFImpl, and ensure it works as before.

[Acfboy]

Thank you for the feedback @2010YOUY01 ! I'd love to try refactoring one operator first to see how it works with the ScalarUDFImpl framework.

I’m thinking of starting with the Bitwise XOR ^ operator for this pilot. It’s limited to integers, making the logic straightforward. Since its type handling is currently in physical execution (not in logical plan as expected), refactoring it will help us understand the full end-to-end flow.

I'm happy to drive this effort. Would you be willing to review my initial PR and provide some support if I hit any roadblocks?

[Omega359]

Interesting idea. I assume you mean to transform 2.0 << 3.5 to something like bit_shift_left(...) and 1 + 'a' to add(...) ?

[Acfboy]

Interesting idea. I assume you mean to transform 2.0 << 3.5 to something like bit_shift_left(...) and 1 + 'a' to add(...) ?

Yes, that's exactly what I mean.
The big difference between bitwise operation handling and others in above examples is likely a legacy of prioritizing vectorized kernel performance in earlier stages. However, it seems we can refactor this into a function-based approach while maintaining the same performance within the current ScalarUDF framework

[2010YOUY01]

I'm happy to drive this effort. Would you be willing to review my initial PR and provide some support if I hit any roadblocks?

Sure, this will be very helpful. And once I see the draft, I’ll have a better understanding for deciding the next steps.

[Acfboy]

take