fix: Emit one zlib stream per compressed debug section - #2162
Conversation
14f975f to
e8a6cfd
Compare
|
The output of the |
davidlattimore
left a comment
There was a problem hiding this comment.
Thanks for fixing this!
Do you think it'd be worth adding a unit test that creates some data, zlib compresses it, concatenates the chunks, decompresses it and then checks that we get back to what we started with?
| let mut output = Vec::new(); | ||
| zstd::stream::copy_encode(chunk, &mut output, ZSTD_COMPRESSION_LEVEL)?; | ||
| Ok(output) | ||
| impl SectionCompressor for ZstdCompressor { |
There was a problem hiding this comment.
Are tools that consume zstd OK with multiple, independently compressed chunks? i.e. do we need to (in a separate PR) look to make a similar fix for zstd, or is it already OK?
There was a problem hiding this comment.
In zstd, a sequence of concatenated frames is a valid format (yes, I should have mentioned this from the beginning 😅):
Zstandard compressed data is made up of one or more frames. Each frame is independent and can be decompressed independently of other frames. The decompressed content of multiple concatenated frames is the concatenation of each frame's decompressed content.
Previously, large compressed debug sections could be emitted incorrectly once a section was split into multiple compression chunks (chunk size is at least 64 KiB). Consumers then saw truncated or corrupt DWARF. (Also see #2119 (comment))
For example, with a buggy wild-linked binary:
whereas GNU ld on the same inputs (note that there is no warning and the number of functions is very different from wild's case):
The root cause was concatenating complete zlib streams from
ZlibEncoder::finish()per chunk. We now compress raw deflate shards in parallel withZ_SYNC_FLUSHboundaries, then emit a single zlib stream (header + shards + empty stored block + combined Adler-32).