Skip to content

GCC LTO merges initializers of ISA-flagged files into one load-time constructor (SIGILL in OpenEXR 3.5.3) #416

Open

Description

@bim-ba

OpenEXR 3.5.3 built with its bundled OpenJPH and GCC LTO crashes with SIGILL as soon as the library is loaded, on CPUs without AVX-512. Arch Linux shipped such a build (openexr 3.5.3-2). On my machine (Core Ultra 9 185H) it killed the SDDM login screen (the crash was in its image-loading thread), and Dolphin crashed the same way.

Cause

With LTO, GCC merges all static initializers into one constructor, _sub_I_65535_0. The merge includes the initializers of the files that src/core/CMakeLists.txt compiles with -mavx2 or -mavx512f -mavx512cd.

The merged constructor then contains AVX-512 instructions. The first one to fault is vmovdqu64 %zmm0,0x2(%rbx). The constructor runs when the library is loaded, before any CPU check.

Reproduction

OpenEXR v3.5.3 with the bundled OpenJPH, GCC 16.2.1, Arch makepkg flags. Test: LD_PRELOAD=libOpenEXRCore-3_5.so /bin/true on a CPU without AVX-512.

Build zmm instructions in constructor VEX/EVEX instructions in constructor Result
-flto=auto 32 95 SIGILL
no LTO 0 0 loads
-flto=auto, OJPH_DISABLE_AVX512=ON 0 61 loads, but still needs AVX/AVX2

Standalone OpenJPH

A standalone LTO build of master (0.33.0 plus 345a8bc) loads fine. Its merged constructor also contains initializers from the ISA-flagged files (for example uvlc_init_tables), but it has no vector instructions. The crash only appears when another project's initializers are merged in, as in OpenEXR.

I have not checked whether this is a known GCC bug (merged constructors should not inherit one file's target options). GCC Bugzilla was blocked for me.

Fix

Compile the ISA-flagged files with -fno-lto. The pull request follows in this thread. With the change:

  • OpenEXR v3.5.3 with the patched bundled copy: 0 vector instructions in the constructor, the library loads, and the AVX-512 kernels stay in the library for runtime dispatch.
  • OpenJPH master, -flto=auto: ctest passes 135 of 135, and each ISA file keeps its own initializer function.

Not tested: clang, non-x86 targets, a full HTJ2K round trip with the patched OpenEXR build.

Activity

  1. aous72 commented on Nov 3, 2026

    @aous72
    Owner

    Hello Everyone,

    It seems that this is an evolving issue. So I will wait the outcome.
    I think some people will be unhappy with this, because it removes some function from LTO.
    I do not like LTO, and I think its contribution to OpenJPH is small -- did not check; it causes a lot of headache.

    Cheers,
    Aous.

    PS: #417.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions