Preserve array derivative views in implicit residual code generation

[ChrisRackauckas]

🚧 UNREVIEWED — awaiting review by @ChrisRackauckas. Written by an AI agent running as @ChrisRackauckas; Chris has not reviewed this implementation. Ignore this PR until he reviews it.
Harness: Codex CLI 0.153.4 · Model: gpt-6-astra
Conversation: local session 01a08018-78ff-7e42-a964-f5174f3f763b (no shareable session URL)

Implicit DAE residuals currently expand an array derivative into an expression containing every scalar derivative. Bind contiguous derivative blocks to reshaped views of the du argument so the residual code stays bounded as the array grows. Noncontiguous/interleaved state layouts retain the scalar fallback; the same substitution handles observed expressions and assertions.

Fixes #5097.

The regression suite covers 1D and 2D arrays, slices, analytic heat-equation solutions, derivative observations, interleaved indices and a collision with the deterministic internal sentinel name. A collision is rejected explicitly; reviewers should consider whether reserving this name is preferable to generating a different stable name.

Validation on Julia 1.12.6, aarch64 Linux, ModelingToolkitBase 1.70.0 development tree:

• Executed lib/ModelingToolkitBase/test/array_equation_dae.jl with the unfixed code loaded, then with this implementation. Before:

  array derivative code size: Test Failed
    Evaluated: 623 == 1631
  array derivative code size | 6 passed, 1 failed
  

  After: 38/38 passed across all testsets. The 2D generated residual contains 331 syntax nodes at every tested size.
• Pkg.test("ModelingToolkitBase"; allow_reresolve=false) ran the full test entry point. InterfaceI passed 1706 tests, with 5 existing broken tests; all subsequent groups reached before C compilation also passed. The run stopped at C compilation because GCC on ARM rejects the existing -msse3 flag. Groups after that failure were not reached. This is not a claim that the full suite is green.
• GROUP=QA: targeted JET 54/54, other QA checks 20 passed/1 failed. Package-wide JET reports 234 possible errors. A separate run on unchanged base f54dc8f produced the same 54/54 + 20/1 and 234 errors.
• Documentation build exited 0. Runic, changed-file typos, and git diff --check passed.

Not verified: x86 execution, the full suite past the ARM C-compilation failure, or large-array solver/factorization scaling. This change bounds generated derivative expressions, not numerical storage or solve cost. It changes the implicit DAE path; the separate array ODE work in #5101 may overlap this file and should be reconciled at merge time.

Follow-up commits (2026-09-12)

Two commits were added on top of 7bb83a3 (harness: Claude Code 2.1.270 · model: claude-fable-5-1; local session 9b5c3ae5-6a0f-41b8-bd69-03bf3b5e7083, no shareable URL):

• e19beae — Handle observations of differentiated array slices. build_explicit_observed_function collects the differentiated variables of every isdiffeq equation into dervars; when an equation's LHS is an array-shaped derivative (a D(u[2:(n-1)]) slice, as the array residual path now preserves), pushing it into dervars broke observation of expressions over that array. The fix skips array-shaped LHSs there (SU.is_array_shape(SU.shape(eq.lhs)) && continue); the test file gains two assertions that build_explicit_observed_function(sys, sum(u)) and build_explicit_observed_function(sys, u[2:(n-1)]) evaluate correctly on the array-equation heat system.
• df59970 — Require DiffEqBase 7.18.1. The new test solves the DAEProblem with DFBDF() and initializealg = BrownFullBasicInit(). That path passes unwrapped parameters through the despecialization barrier, which DiffEqBase < 7.18.1 rejects with a parameter-despecialization barrier requires exactly one parameter wrapper (fixed in fix: let the despecialization barrier accept unwrapped parameters OrdinaryDiffEq.jl#4297). The downgrade-mtkbase (InterfaceI) job resolved DiffEqBase 7.15.0 and hit exactly that error (https://github.com/SciML/ModelingToolkit.jl/actions/runs/34698852412/job/103567159695). DiffEqBase = "7.14" is raised to "7.18.1" in lib/ModelingToolkitBase/Project.toml and, for consistency, in the top-level Project.toml.

Validation of the final head (Julia 1.12.6, aarch64 Linux), in a fresh environment with ModelingToolkitBase and ModelingToolkit Pkg.developed from this branch and DiffEqBase resolved to 7.21.1:

• include("lib/ModelingToolkitBase/test/array_equation_dae.jl") in a fresh process:

  DiffEqBase version: 7.21.1
  Test Summary:                  | Pass  Total     Time
  focused: array_equation_dae.jl |   40     40  1m48.8s
  EXIT 0
  

• Runic 1.10.0 --check --diff on the three changed .jl files: clean. typos 1.50.1 over all five changed files with the repo .typos.toml: clean. git diff --check: clean.

Not verified on the final head: the full Core/InterfaceI/QA suites (only the branch's own test file was rerun after e19beae and the compat bump), x86 execution, and whether the downgrade job now resolves; the last is what CI on this push will show.

---

🤖 Posted by an AI agent — harness: Codex CLI 0.153.4 · model: gpt-6-astra
Conversation: local session 01a08018-78ff-7e42-a964-f5174f3f763b (no shareable session URL)

[github-actions]

Benchmark Results (Julia v1)

Time benchmarks

	master	df59970...	master / df59970...
ODEProblem	5.08 ± 0.15 ms	5.1 ± 0.27 ms	0.996 ± 0.061
init	0.0404 ± 0.024 ms	0.0405 ± 0.022 ms	0.997 ± 0.8
large_parameter_init/ODEProblem	14.4 ± 0.9 ms	14.1 ± 0.7 ms	1.02 ± 0.082
large_parameter_init/init	0.0659 ± 0.0086 ms	0.0658 ± 0.0054 ms	1 ± 0.15
mtkcompile	8.87 ± 0.95 ms	9.13 ± 0.8 ms	0.972 ± 0.13
sparse_analytical_jacobian/ODEProblem	18.7 ± 2.3 ms	19.5 ± 1.8 ms	0.963 ± 0.15
sparse_analytical_jacobian/f_iip	0.07 ± 0 μs	0.07 ± 0 μs	1 ± 0
sparse_analytical_jacobian/f_oop	0.109 ± 0.015 ms	0.108 ± 0.014 ms	1.01 ± 0.19
time_to_load	5.15 ± 0.061 s	5.13 ± 0.054 s	1 ± 0.016

Memory benchmarks

	master	df59970...	master / df59970...
ODEProblem	0.0377 M allocs: 1.7 MB	0.0377 M allocs: 1.7 MB	1
init	0.455 k allocs: 0.0589 MB	0.455 k allocs: 0.0589 MB	1
large_parameter_init/ODEProblem	0.157 M allocs: 5.4 MB	0.157 M allocs: 5.4 MB	1
large_parameter_init/init	0.838 k allocs: 0.171 MB	0.838 k allocs: 0.171 MB	1
mtkcompile	0.0557 M allocs: 2.62 MB	0.0557 M allocs: 2.62 MB	1
sparse_analytical_jacobian/ODEProblem	0.117 M allocs: 4.76 MB	0.117 M allocs: 4.76 MB	1
sparse_analytical_jacobian/f_iip	0 allocs: 0 B	0 allocs: 0 B
sparse_analytical_jacobian/f_oop	0.848 k allocs: 27 kB	0.848 k allocs: 27 kB	1
time_to_load	0.201 k allocs: 11.8 kB	0.201 k allocs: 11.8 kB	1

[github-actions]

Benchmark Results (Julia vlts)

Time benchmarks

	master	df59970...	master / df59970...
ODEProblem	5.71 ± 0.11 ms	5.73 ± 0.13 ms	0.996 ± 0.029
init	0.102 ± 0.036 ms	0.0791 ± 0.039 ms	1.29 ± 0.79
large_parameter_init/ODEProblem	17.1 ± 0.5 ms	17 ± 0.45 ms	1 ± 0.04
large_parameter_init/init	0.101 ± 0.044 ms	0.103 ± 0.044 ms	0.978 ± 0.6
mtkcompile	10 ± 0.38 ms	9.91 ± 0.26 ms	1.01 ± 0.046
sparse_analytical_jacobian/ODEProblem	20.8 ± 0.56 ms	20.9 ± 0.62 ms	0.994 ± 0.04
sparse_analytical_jacobian/f_iip	0.09 ± 0.01 μs	0.08 ± 0.01 μs	1.12 ± 0.19
sparse_analytical_jacobian/f_oop	0.388 ± 0.013 ms	0.388 ± 0.011 ms	0.999 ± 0.045
time_to_load	6.84 ± 0.0097 s	6.86 ± 0.032 s	0.997 ± 0.0049

Memory benchmarks

	master	df59970...	master / df59970...
ODEProblem	0.0375 M allocs: 1.95 MB	0.0375 M allocs: 1.95 MB	1
init	0.5 k allocs: 0.0843 MB	0.5 k allocs: 0.0843 MB	1
large_parameter_init/ODEProblem	0.161 M allocs: 6.06 MB	0.161 M allocs: 6.07 MB	1
large_parameter_init/init	0.682 k allocs: 0.185 MB	0.682 k allocs: 0.185 MB	1
mtkcompile	0.0594 M allocs: 3.38 MB	0.0594 M allocs: 3.36 MB	1
sparse_analytical_jacobian/ODEProblem	0.118 M allocs: 5.73 MB	0.118 M allocs: 5.73 MB	1
sparse_analytical_jacobian/f_iip	0 allocs: 0 B	0 allocs: 0 B
sparse_analytical_jacobian/f_oop	0.634 k allocs: 19.6 kB	0.634 k allocs: 19.6 kB	1
time_to_load	0.153 k allocs: 14.5 kB	0.153 k allocs: 14.5 kB	1

[ChrisRackauckas]

🤖 Automated comment from an AI agent running as @ChrisRackauckas — not written or reviewed by Chris.

Pushed two commits on top of 7bb83a3 (7bb83a3..df59970, plain push, no force):

• e19beae — Handle observations of differentiated array slices. build_explicit_observed_function now skips array-shaped derivative LHSs when collecting dervars, so observed expressions over an array whose interior is a single slice equation (sum(u), u[2:(n-1)]) evaluate correctly. Adds two assertions to lib/ModelingToolkitBase/test/array_equation_dae.jl.
• df59970 — Require DiffEqBase 7.18.1. The new test's solve(::DAEProblem, DFBDF(); initializealg = BrownFullBasicInit()) needs the despecialization-barrier fix from fix: let the despecialization barrier accept unwrapped parameters OrdinaryDiffEq.jl#4297; downgrade-mtkbase (InterfaceI) resolved DiffEqBase 7.15.0 and failed with a parameter-despecialization barrier requires exactly one parameter wrapper (https://github.com/SciML/ModelingToolkit.jl/actions/runs/34698852412/job/103567159695). DiffEqBase = "7.14" → "7.18.1" in lib/ModelingToolkitBase/Project.toml and the top-level Project.toml.

Observed on the final head (Julia 1.12.6, aarch64 Linux; fresh env with ModelingToolkitBase and ModelingToolkit Pkg.developed from this branch, DiffEqBase resolved to 7.21.1), running only lib/ModelingToolkitBase/test/array_equation_dae.jl in a fresh process:

Test Summary:                  | Pass  Total     Time
focused: array_equation_dae.jl |   40     40  1m48.8s

Runic 1.10.0 --check --diff on the three changed .jl files, typos 1.50.1 on all five changed files, and git diff --check all clean.

Not verified: the full Core/InterfaceI/QA suites on this head, x86, and whether the downgrade job now resolves — the CI run on this push is the check for that.

---

🤖 Posted by an AI agent — harness: Claude Code 2.1.270 · model: claude-fable-5-1
Conversation: local Claude Code session 9b5c3ae5-6a0f-41b8-bd69-03bf3b5e7083 (no shareable URL available)

[ChrisRackauckas-Claude]

This PR's commit has been carried forward, with its authorship kept, onto current master in #5242. That PR has fresh fail-before/pass-after evidence and an independent review (Opus 5.5: MERGE, low). If #5232 is the one to merge, this one can be closed.

🤖 Generated with Claude Code (model: claude-opus-5-5[1m]), https://claude.ai/code/session_01H4fXBChQshAwfpLMCpEvAX