feat: Add FFI_QueryPlanner to support foreign query planners across shared-library boundaries

[milenkovicm]

Which issue does this PR close?

This is follow up on #20249 which was co-authored with @timsaucer

Closes #. (pending)

Rationale for this change

DataFusion's FFI layer allows plugins and foreign libraries to extend query processing across shared-library boundaries. However, there was no way for a foreign library to participate in the physical planning pipeline — QueryPlanner required a concrete &SessionState, which cannot cross an FFI boundary.

What changes are included in this PR?

• Adds FFI_QueryPlanner — an FFI-safe repr-C struct that wraps a foreign QueryPlanner implementation, serializing the LogicalPlan via datafusion-proto across the boundary and deserializing the resulting ExecutionPlan.
• Adds QueryPlannerWeak trait — a variant of QueryPlanner that accepts &dyn Session instead of &SessionState, making it implementable by foreign code. It make more sense to create new interface rather than changing QueryPlanner. The change would introduce big backward incompatibility, also we can argue that having SessionState as parameter makes sense.
• Extends QueryPlanner with an Any bound to allow downcasting (needed by DynamicForeignQueryPlanner).
• Adds integration tests covering both direct use of FFI_QueryPlanner via QueryPlannerWeak and the deferred-injection pattern via DynamicForeignQueryPlanner.

The DynamicForeignQueryPlanner helper (in the integration tests) shows how to break the construction-time dependency cycle: SessionContext needs a QueryPlanner at build time, but the FFI planner needs the codec which needs the context. The placeholder is registered at build time and swapped for the real FFI planner once both sides exist.

Are these changes tested?

Yes. Integration tests are added in datafusion/ffi/tests/ffi_planner.rs:

• test_ffi_query_planner — verifies that a foreign planner loaded from a shared library can produce a valid ExecutionPlan from a LogicalPlan.
• test_ffi_dynamic_query_planner — verifies the deferred-injection pattern where the FFI planner is installed into an already-constructed SessionContext via DynamicForeignQueryPlanner.

Are there any user-facing changes?

Yes, one change to a public trait:

• QueryPlanner now requires Any as a supertrait (enables downcasting). This is not required for this implementation but it might be needed for implementors

Open Questions

• there is chicken-egg problem at the moment, SessionContext is needed to create a QueryPlanner but QueryPlanner is required to create a SessionContext (I hope I did not get that horrible wrong)
• Name of QueryPlannerWeak is debatable, I cant come up with better name

[github-actions]

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion v53.1.0 (current)
       Built [ 102.481s] (current)
     Parsing datafusion v53.1.0 (current)
      Parsed [   0.036s] (current)
    Building datafusion v53.1.0 (baseline)
       Built [ 102.120s] (baseline)
     Parsing datafusion v53.1.0 (baseline)
      Parsed [   0.036s] (baseline)
    Checking datafusion v53.1.0 -> v53.1.0 (no change; assume patch)
     Checked [   0.635s] 222 checks: 221 pass, 1 fail, 0 warn, 30 skip

--- failure trait_added_supertrait: non-sealed trait added new supertraits ---

Description:
A non-sealed trait added one or more supertraits, which breaks downstream implementations of the trait
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#generic-bounds-tighten
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.47.0/src/lints/trait_added_supertrait.ron

Failed in:
  trait datafusion::execution::context::QueryPlanner gained Any in file /home/runner/work/datafusion/datafusion/datafusion/core/src/execution/context/mod.rs:2077

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [ 206.888s] datafusion
    Building datafusion-ffi v53.1.0 (current)
       Built [  58.568s] (current)
     Parsing datafusion-ffi v53.1.0 (current)
      Parsed [   0.059s] (current)
    Building datafusion-ffi v53.1.0 (baseline)
       Built [  59.116s] (baseline)
     Parsing datafusion-ffi v53.1.0 (baseline)
      Parsed [   0.058s] (baseline)
    Checking datafusion-ffi v53.1.0 -> v53.1.0 (no change; assume patch)
     Checked [   0.229s] 222 checks: 220 pass, 1 fail, 1 warn, 30 skip

--- failure constructible_struct_adds_field: externally-constructible struct adds field ---

Description:
A pub struct constructible with a struct literal has a new pub field. Existing struct literals must be updated to include the new field.
        ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.47.0/src/lints/constructible_struct_adds_field.ron

Failed in:
  field ForeignLibraryModule.create_query_planner in /home/runner/work/datafusion/datafusion/datafusion/ffi/src/tests/mod.rs:105

--- warning repr_c_plain_struct_fields_reordered: struct fields reordered in repr(C) struct ---

Description:
A public repr(C) struct had its fields reordered. This can change the struct's memory layout, possibly breaking FFI use cases that depend on field position and order.
        ref: https://doc.rust-lang.org/reference/type-layout.html#reprc-structs
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.47.0/src/lints/repr_c_plain_struct_fields_reordered.ron

Failed in:
  ForeignLibraryModule.version moved from position 15 to 16, in /home/runner/work/datafusion/datafusion/datafusion/ffi/src/tests/mod.rs:108

     Summary semver requires new major version: 1 major and 0 minor checks failed
     Warning produced 1 major and 0 minor level warnings
    Finished [ 119.378s] datafusion-ffi