Maintenance

This release is primarily a maintenance release.

It includes a few bug fixes, including a fix for a rare data corruption bug, which could only be triggered in a niche use case, when doing all of the following: using multithreading mode, with an overlap size >= 512 MB, using a strategy >= ZSTD_btlazy, and compressing more than 4 GB. None of the default compression levels meet these requirements (not even --ultra ones).

Performance

This release also includes some performance improvements, among which the primary improvement is that Zstd decompression is ~7% faster, thanks to @mgrice.

See this comparison of decompression speeds at different compression levels, measured on the Silesia Corpus, on an Intel i9-9900K with GCC 9.1.0.

Detailed list of changes

• bug: Fix data corruption in niche use cases (huge inputs + multithreading + large custom window sizes + other conditions) by @terrelln (#1659)
• bug: Fuzz legacy modes, fix uncovered bugs by @terrelln (#1593, #1594, #1595)
• bug: Fix out of bounds read by @terrelln (#1590)
• perf: Improved decoding speed by ~7% @mgrice (#1668)
• perf: Large compression ratio improvement for small windowLog by @cyan4973 (#1624)
• perf: Faster compression speed in high compression mode for repetitive data by @terrelln (#1635)
• perf: Slightly improved compression ratio of level 3 and 4 (ZSTD_dfast) by @cyan4973 (#1681)
• perf: Slightly faster compression speed when re-using a context by @cyan4973 (#1658)
• api: Add parameter to generate smaller dictionaries by @tyler-tran (#1656)
• cli: Recognize symlinks when built in C99 mode by @felixhandte (#1640)
• cli: Expose cpu load indicator for each file on -vv mode by @ephiepark (#1631)
• cli: Restrict read permissions on destination files by @chungy (#1644)
• cli: zstdgrep: handle -f flag by @felixhandte (#1618)
• cli: zstdcat: follow symlinks by @vejnar (#1604)
• doc: Remove extra size limit on compressed blocks by @felixhandte (#1689)
• doc: Improve documentation on streaming buffer sizes by @cyan4973 (#1629)
• build: CMake: support building with LZ4 @leeyoung624 (#1626)
• build: CMake: install zstdless and zstdgrep by @leeyoung624 (#1647)
• build: CMake: respect existing uninstall target by @j301scott (#1619)
• build: Make: skip multithread tests when built without support by @michaelforney (#1620)
• build: Make: Fix examples/ test target by @sjnam (#1603)
• build: Meson: rename options out of deprecated namespace by @lzutao (#1665)
• build: Meson: fix build by @lzutao (#1602)
• build: Visual Studio: don't export symbols in static lib by @scharan (#1650)
• build: Visual Studio: fix linking by @absotively (#1639)
• build: Fix MinGW-W64 build by @myzhang1029 (#1600)
• misc: Expand decodecorpus coverage by @ephiepark (#1664)