Skip to content

Core: byte-swap the float vector data, not the attribute struct (fixes big-endian crash)#2510

Merged
cary-ilm merged 2 commits into
AcademySoftwareFoundation:mainfrom
Scottcjn:bugfix/be-float-vector-attr
Jul 4, 2026
Merged

Core: byte-swap the float vector data, not the attribute struct (fixes big-endian crash)#2510
cary-ilm merged 2 commits into
AcademySoftwareFoundation:mainfrom
Scottcjn:bugfix/be-float-vector-attr

Conversation

@Scottcjn

Copy link
Copy Markdown
Contributor

Addresses the testAttributes failure in #1175.

extract_attr_float_vector reads the float array into attrdata->arr, then byte-swaps with priv_to_native32(attrdata, n). On big-endian hosts that swaps the first n 32-bit words of the exr_attr_float_vector_t struct (the arr pointer and length) instead of the n floats the array holds. The corrupted pointer and length then crash when the attribute is read back: OpenEXR.testAttributes segfaults constructing the FloatVector in Context::header (a std::vector<float> built from a garbage pointer range with a ~64M element count).

The fix swaps attrdata->arr instead. priv_to_native32 is a no-op on little-endian, so x86_64 and ppc64le are unaffected.

The swap runs only inside if (rv == EXR_ERR_SUCCESS && n > 0), after exr_attr_float_vector_init has allocated attrdata->arr and sequential_read has filled it, so attrdata->arr is always valid and non-null at this point.

Testing

On real POWER8 hardware (RelWithDebInfo build, crash localized with gdb):

  • ppc64 big-endian (gcc 13.3): OpenEXR.testAttributes segfaults before this change and passes after.
  • Little-endian (x86_64, ppc64le) is unchanged: the swapped call does nothing on little-endian, so any EXR with a float-vector attribute reads identically there.

extract_attr_float_vector reads the float array into attrdata->arr, then
called priv_to_native32(attrdata, n). On big-endian hosts that swaps the
first n 32-bit words of the exr_attr_float_vector_t struct itself (the
arr pointer and length) instead of the n floats in attrdata->arr. The
corrupted pointer and length then crash when the value is read back, for
example a std::bad_alloc or segfault while constructing the FloatVector
in the C++ Context::header path.

Swap attrdata->arr instead. priv_to_native32 is a no-op on little-endian,
so x86_64 and ppc64le are unaffected.

Verified on POWER8: OpenEXR.testAttributes segfaults on big-endian ppc64
before this change and passes after.

Signed-off-by: Scott Boudreaux <121303252+Scottcjn@users.noreply.github.com>
@Scottcjn

Copy link
Copy Markdown
Contributor Author

Quick heads-up: the unstable state here is not a CI failure. The GitHub Actions workflows are sitting at action_required (awaiting maintainer approval to run), not red. The sibling PR #2509 had its workflows approved and passed clean. If a maintainer can approve the run here and take a look, this should green up the same way. The change is the one-line byte-swap-target fix in extract_attr_float_vector (swap the float array, not the attribute struct header), validated on a native big-endian ppc64 host where testAttributes segfaults before the fix and passes after.

@cary-ilm

Copy link
Copy Markdown
Member

Thanks! I poked the CI and everything passes. It looks good to me but I'd like @kdt3rd to look at it before merging. Thanks for the fix. I remember that issue coming in ages ago but wasn't sure what to do about it, so we really appreciate the help.

@Scottcjn

Copy link
Copy Markdown
Contributor Author

Thanks for the opportunity to use my weird knowledgebase and skillsets to help ilm.
😆

@kdt3rd kdt3rd left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm - thanks for noticing that!

@cary-ilm
cary-ilm merged commit e198e44 into AcademySoftwareFoundation:main Jul 4, 2026
2 of 3 checks passed
@cary-ilm cary-ilm added the v3.4 label Jul 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants