Memory stomp due to libdeflate_gdeflate_compress_bound returning too small of a value and libdeflate_gdeflate_compress ignoring the output limit

[elasota]

While running some tests, GDeflateDemo crashed due to stack corruption. Upon testing, it turns out that this was due to libdeflate_gdeflate_compress writing more data than libdeflate_gdeflate_compress_bound indicated was the upper limit.

Reproduction steps:

• Extract "gdeflatecrash.bin" from the attached file
• Compile GDeflateDemo in debug mode so stack corruption checks are enabled
• Run GDeflateDemo /compress gdeflatecrash.bin .

gdeflatecrash.zip

[elasota]

I guess this is actually a 2-part bug: libdeflate_gdeflate_compress_bound returns too small of a value, but libdeflate_gdeflate_compress is also supposed to fail safely if more output is written than is specified in the page buffer, which it sometimes does, but with this sample, it is stomping memory above the provided limit.

[coopp]

Thanks for reporting this issue. I will take this issue as a bug so we can track it accordingly.

[larryche]

This PR updated libdeflate. Can you check if the issue continues to repro with that?