DiffEqBase: de-specialize p via OpaqueVoid in AutoSpecialize

[ChrisRackauckas-Claude]

Note

Draft PR — please ignore until reviewed by @ChrisRackauckas.

Important

Depends on SciML/RespecializeParams.jl being registered in the General registry. The new lib/DiffEqBase/Project.toml [deps] entry points at RespecializeParams 0.1. Once registered, this PR's CI should pick it up.

Summary

Inside lib/DiffEqBase, when promote_f decides to wrap an in-place ODE rhs under AutoSpecialize / FunctionWrapperSpecialize, and p is an isbits non-NullParameters payload:

1. The Void(f.f) carrier is replaced with a new DiffEqBase.OpaqueVoid{P, F} that holds the concrete parameter type P as a type parameter alongside the user's RHS.
2. The FunctionWrappersWrapper signature(s) — for f.f, f.tgrad, and f.jac — get RespecializeParams.OpaqueParams substituted in place of typeof(p).
3. p is packed into an OpaqueParams and returned alongside the wrapped f from promote_f. get_concrete_problem then sets prob.p to the opaque container.

At call time OpaqueVoid{P}(args...) does a single RespecializeParams.unsafe_unpack(op, P) and forwards (du, u, p::P, t) (or the 3-arg form) to the user's f. The unpack is type-stable and allocation-free.

Net effect: two ODEProblems whose parameter struct types differ (LorenzP_v1, LorenzP_v2, ...) now share the same precompiled FunctionWrappersWrapper / Void / integrator specialization, since typeof(prob.p) === OpaqueParams for both. The user's f is still fully specialized on the original concrete P since P flows through OpaqueVoid's type parameter.

NullParameters and non-isbits payloads keep the existing Void wrapping path unchanged.

What changed

• lib/DiffEqBase/Project.toml — adds RespecializeParams = 0.1 to [deps] + [compat].
• lib/DiffEqBase/src/opaque_void.jl — new file: OpaqueVoid, should_opaque_p, pack_p_for_opaque, wrap_void_opaque, wrapfun_iip_opaque (single-variant; the FD multi-variant version lives in the extension).
• lib/DiffEqBase/src/DiffEqBase.jl — import RespecializeParams and include("opaque_void.jl").
• lib/DiffEqBase/src/solve.jl — refactors both promote_f overloads (Val{true} = FD path, Val{false} = no-FD path) to return (f, p) instead of just f, threads the opaque branch through, and updates the two get_concrete_problem callers to unpack the tuple and remake with p_promote. The SplitFunction overloads also return (f, p). promote_f is DiffEqBase-internal — no extensions override it — so the signature change is private.
• lib/DiffEqBase/ext/DiffEqBaseForwardDiffExt.jl — adds wrapfun_iip_opaque(ff, ::Type{P}, inputs, Val(CS)) mirroring the existing 4-variant wrapfun_iip (plain / Jacobian-Dual / time-Dual / both), reusing the _make_fww inference shim. Stiff algorithms that call the wrapped RHS with Dual u for Jacobian and/or Dual t for tgrad find a matching variant.
• lib/DiffEqBase/test/opaque_p_test.jl — new test, exercises the path through get_concrete_problem directly (no integrator dep): isbits → opaque, FullSpecialize → no-op, NullParameters → no-op, non-isbits → no-op, two problems with different LinearP values share typeof(prob.f) and typeof(prob.p), unsafe_unpack(cp.p, LinearP) roundtrips.
• lib/DiffEqBase/test/runtests.jl — registers the new safetestset in the Core group.

Test plan

• Local Core tests pass. GROUP=Core Pkg.test() on lib/DiffEqBase is clean (8/8 new opaque tests + the rest of the Core suite unchanged).
• End-to-end against OrdinaryDiffEqTsit5 (Val{false} wrapping path). Lorenz with LorenzP → numerical result matches FullSpecialize baseline to 1e-9; sol.prob.p isa OpaqueParams; two LorenzP problems share typeof(sol.prob.f).
• End-to-end against OrdinaryDiffEqRosenbrock (Val{true} / ForwardDiff path). Rosenbrock23 on the same Lorenz — converges, matches baseline to 1e-8, sol.prob.p isa OpaqueParams.
• Zero-allocation in the inner loop. A direct call to sol.prob.f.f(du, u, sol.prob.p, t) (i.e. FunctionWrappersWrapper → OpaqueVoid → unsafe_unpack → user f) measures @allocated == 0.
• CI: Tests.yml green across the matrix.
• CI: Downstream.yml green (this is the one I'm most curious about — it's where any breakage from typeof(prob.p) changing under AutoSpecialize would surface).
• CI: FormatCheck.yml — Runic was applied locally on the modified files.

Behavioral change worth flagging

This is on by default under AutoSpecialize (the user explicitly chose that variant in design). After get_concrete_problem runs, the integrator sees prob.p :: RespecializeParams.OpaqueParams instead of the user's original struct. Downstream code that introspects sol.prob.p and expects the original concrete type will need to call RespecializeParams.unpack(sol.prob.p, P) to recover it. SymbolicIndexingInterface lookups, ModelingToolkit, sensitivity adjoints, etc. may be affected. The risk surface is real, which is why this is opening as draft for a careful review before merge.

Out-of-scope for this PR (happy to add in follow-ups):

• OpaqueRef-backed path for non-isbits payloads (e.g. Vector{Float64} parameters). RespecializeParams already exposes the container; just needs the corresponding OpaqueRef-aware wrapfun variants here.
• Adding a way to retrieve the original concrete p from prob without explicitly knowing the payload type (e.g. an accessor that uses the objectid typeid for safety).
• DAE / DDE / SDE paths — currently only ODE promote_f is touched.

[oscardssmith]

I'm fairly skeptical of this being a good idea.