Enzyme Julia 1 job aborts in IIP wrapper mutated-after-call regression #67

Activity

1. ChrisRackauckas-Claude commented on Jul 11, 2026

   

   ChrisRackauckas-Claude

   on Jul 11, 2026

   MemberAuthor

   More actions

   Fresh local check while auditing PR #68:

   • 9c11a85 (v1.9.2) passes timeout 3600 ~/.juliaup/bin/julia +1.12 --startup-file=no --project=. -e 'using Pkg; Pkg.instantiate(); Pkg.test(; test_args=["Enzyme"])' with Enzyme group 65 pass.\n- 103e9fb (Split reverse-mode thunk for the IIP FWW Enzyme rule) fails the same command with signal 6 / Enzyme GradientUtils.cpp:656 assertion at test/Enzyme/enzyme_tests.jl:349.\n- 8eacfee also fails with the same assertion.\n\nSo PR Use SciMLTesting public API docs QA #68 is not introducing this; it is a pre-existing main blocker tracked here.
2. ChrisRackauckas-Claude commented on Jul 12, 2026

   

   ChrisRackauckas-Claude

   on Jul 12, 2026

   MemberAuthor

   More actions

   Additional local triage from a clean origin/main checkout (fd02a35, branch agent/enzyme-main-failure), using the requested command form:

   timeout 3600 env GROUP=Enzyme ~/.juliaup/bin/julia +release --project=. -e 'using Pkg; Pkg.test()'

   Result: reproduced the same signal-6 abort at test/Enzyme/enzyme_tests.jl:349 with Julia 1.12.5, Enzyme v0.13.181, EnzymeCore v0.8.21, Enzyme_jll v0.0.281+0, LLVM v9.10.0, and GPUCompiler v1.23.0.

   Dependency / version boundary checks on the minimal args mutated after call scenario:

   • Julia 1.11.9 passes with the same Enzyme v0.13.181 / Enzyme_jll v0.0.281+0 stack and returns g = [3.1499999999999995, 12.8], matching [4.5p1, 32p2].
   • Julia 1.12.0 fails with the same GradientUtils.cpp:656 assertion and the same Enzyme stack.
   • Julia 1.12.5 also fails; pinning Enzyme v0.13.180 / Enzyme_jll v0.0.280+0 still fails, so this is not just the latest Enzyme release.
   • The same loss written without FunctionWrappersWrapper passes on Julia 1.12.5 with Enzyme v0.13.181.

   I also injected the minimal scenario into 77e373f (v1.9.3, before 103e9fb) under Julia 1.12.5 / Enzyme 0.13.181, and it aborts the same way. So 103e9fb appears to be the first commit that adds suite coverage for this path, not necessarily the root cause of the compiler abort.

   Attempted local workaround: adding Enzyme rules for raw FunctionWrappers.FunctionWrapper analogous to the existing FunctionWrappersWrapper rules did not change the failure; Enzyme still reached jlcapi_CallWrapper IR and aborted. I reverted that experiment locally.

   Current best culprit: Julia 1.12 changes expose an Enzyme failure on the FunctionWrappers call ABI (jlcapi_CallWrapper) used by this wrapped IIP reverse-mode path. A repo-local fix was not identified in this pass.
3. ChrisRackauckas-Claude commented on Jul 15, 2026

   

   ChrisRackauckas-Claude

   on Jul 15, 2026 – with ChatGPT Codex Connector

   MemberAuthor

   More actions

   Formal follow-up from the CI investigation:

   • With Julia 1.12.6 and Enzyme 0.13.183, a source-history bisect identifies 103e9fb692536f06ee10838b712596ad76dac284 (the split reverse-mode thunk change) as the first FunctionWrappersWrappers commit that exercises the aborting path.
   • Its immediate parent 66c60cfd6d95906c7e3297d7aa5896c99dccb750 passes the complete Enzyme test file under the same pinned environment.
   • Current main aborts with Enzyme 0.13.183 and 0.13.184, while Enzyme 0.13.185 passes the full Enzyme group (68/68).

   Draft PR #71 raises both package and test-environment compatibility floors to Enzyme 0.13.185 without changing the regression test.
4. ChrisRackauckas

   closed this as completedin #71on Jul 15, 2026