Core: byte-swap the float vector data, not the attribute struct (fixes big-endian crash)#2510
Conversation
extract_attr_float_vector reads the float array into attrdata->arr, then called priv_to_native32(attrdata, n). On big-endian hosts that swaps the first n 32-bit words of the exr_attr_float_vector_t struct itself (the arr pointer and length) instead of the n floats in attrdata->arr. The corrupted pointer and length then crash when the value is read back, for example a std::bad_alloc or segfault while constructing the FloatVector in the C++ Context::header path. Swap attrdata->arr instead. priv_to_native32 is a no-op on little-endian, so x86_64 and ppc64le are unaffected. Verified on POWER8: OpenEXR.testAttributes segfaults on big-endian ppc64 before this change and passes after. Signed-off-by: Scott Boudreaux <121303252+Scottcjn@users.noreply.github.com>
|
Quick heads-up: the unstable state here is not a CI failure. The GitHub Actions workflows are sitting at action_required (awaiting maintainer approval to run), not red. The sibling PR #2509 had its workflows approved and passed clean. If a maintainer can approve the run here and take a look, this should green up the same way. The change is the one-line byte-swap-target fix in extract_attr_float_vector (swap the float array, not the attribute struct header), validated on a native big-endian ppc64 host where testAttributes segfaults before the fix and passes after. |
|
Thanks! I poked the CI and everything passes. It looks good to me but I'd like @kdt3rd to look at it before merging. Thanks for the fix. I remember that issue coming in ages ago but wasn't sure what to do about it, so we really appreciate the help. |
|
Thanks for the opportunity to use my weird knowledgebase and skillsets to help ilm. |
kdt3rd
left a comment
There was a problem hiding this comment.
lgtm - thanks for noticing that!
Addresses the
testAttributesfailure in #1175.extract_attr_float_vectorreads the float array intoattrdata->arr, then byte-swaps withpriv_to_native32(attrdata, n). On big-endian hosts that swaps the firstn32-bit words of theexr_attr_float_vector_tstruct (thearrpointer andlength) instead of thenfloats the array holds. The corrupted pointer and length then crash when the attribute is read back:OpenEXR.testAttributessegfaults constructing theFloatVectorinContext::header(astd::vector<float>built from a garbage pointer range with a ~64M element count).The fix swaps
attrdata->arrinstead.priv_to_native32is a no-op on little-endian, so x86_64 and ppc64le are unaffected.The swap runs only inside
if (rv == EXR_ERR_SUCCESS && n > 0), afterexr_attr_float_vector_inithas allocatedattrdata->arrandsequential_readhas filled it, soattrdata->arris always valid and non-null at this point.Testing
On real POWER8 hardware (RelWithDebInfo build, crash localized with gdb):
OpenEXR.testAttributessegfaults before this change and passes after.