Make deflateBound() account for a preset dictionary set after the call - #1308
Open
XananasX7 wants to merge 1 commit into
Open
Make deflateBound() account for a preset dictionary set after the call#1308XananasX7 wants to merge 1 commit into
XananasX7 wants to merge 1 commit into
Conversation
zlib.h documents that deflateBound() returns an upper bound on the compressed size such that a first deflate() call given all of the input at once, Z_FINISH, and an output buffer of that bound, is guaranteed to return Z_STREAM_END. deflateSetDictionary() may legally be called after deflateBound() and before that first deflate() (its only requirement is that it precede any deflate() call). When it is, the emitted zlib header carries the PRESET_DICT flag followed by the four bytes of the dictionary's Adler-32 (deflate.c, header emission), but deflateBound() only included those four bytes when s->strstart was already non-zero at bound time, i.e. when the dictionary had been set before deflateBound(). Callers following the documented single-pass pattern therefore received a bound four bytes too small: the first deflate() returned Z_OK with the output buffer full instead of Z_STREAM_END, truncating the stream (the dictionary Adler-32 and the first bytes of compressed data were lost). The compressed stream was still syntactically valid deflate but was no longer the Adler-32-verified stream the caller expected. Fix: include the four-byte preset-dictionary Adler-32 whenever the zlib header has not yet been emitted (stream status is still INIT_STATE), not only when a dictionary has already been loaded. Once the header has been written the bytes are already accounted for in the output. This makes the bound conservative by at most four bytes for streams that never use a dictionary.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
deflateBound()can return a bound that is four bytes too small, breaking the single-pass guarantee documented in zlib.h, whendeflateSetDictionary()is called afterdeflateBound()and before the firstdeflate().The documented contract (zlib.h)
deflateSetDictionary()'s documentation only requires that it be called afterdeflateInit()/deflateInit2()and beforedeflate()— no ordering relative todeflateBound()is stated. So init → bound → setDictionary → single-pass finish is a fully legal sequence.Root cause
When a preset dictionary is in use, the emitted zlib header carries the
PRESET_DICTflag followed by the four bytes of the dictionary's Adler-32 (deflate.c header emission).deflateBound_z()accounts for this only viawraplen = 6 + (s->strstart ? 4 : 0):deflateBound()→s->strstartis non-zero → the +4 is included. Correct.deflateBound()(legal per the docs) → at bound times->strstart == 0and the stream has not emitted its header → +4 is not included → the bound is short by exactly the four dictionary-Adler bytes that the later header will carry.Impact
A caller that allocates exactly
deflateBound()bytes and relies on the documented one-callZ_STREAM_ENDguarantee getsZ_OKwithavail_out == 0instead — the output buffer fills up four bytes early, and the stream is silently truncated (missing the dictionary Adler-32 and the first bytes of compressed data). Downstream decoders either fail or, for streams where the truncation lands on a block boundary, produce a subtly different stream.Verified on current develop (e3dc0a8): for incompressible input of 1–100,000 bytes with dictionary lengths 1–30000, every case returned
Z_OKwith the buffer full instead ofZ_STREAM_END(35/35 configurations).Fix
Include the four preset-dictionary bytes whenever the header has not yet been emitted — i.e. when no data has been compressed yet and the stream status is still
INIT_STATE— not only when a dictionary has already been loaded:Once the header has been emitted (status past
INIT_STATE), the dictionary Adler-32 is already part of the accounted output, so no double counting occurs. The bound becomes conservative by at most four bytes for streams that never use a preset dictionary.Verification
deflateInit→deflateBound(n)→deflateSetDictionary→ one-shotdeflate(Z_FINISH)withavail_out = bound. Before:Z_OK, buffer full (35/35 cases). After:Z_STREAM_ENDin all cases.make testpasses on the patched tree.