MyGit

v1.5.5

facebook/zstd

版本发布时间: 2023-04-05 06:20:32

facebook/zstd最新发布版本:v1.5.6(2024-03-31 02:57:28)

Zstandard v1.5.5 Release Note

This is a quick fix release. The primary focus is to correct a rare corruption bug in high compression mode, detected by @danlark1 . The probability to generate such a scenario by random chance is extremely low. It evaded months of continuous fuzzer tests, due to the number and complexity of simultaneous conditions required to trigger it. Nevertheless, @danlark1 from Google shepherds such a humongous amount of data that he managed to detect a reproduction case (corruptions are detected thanks to the checksum), making it possible for @terrelln to investigate and fix the bug. Thanks !
While the probability might be very small, corruption issues are nonetheless very serious, so an update to this version is highly recommended, especially if you employ high compression modes (levels 16+).

When the issue was detected, there were a number of other improvements and minor fixes already in the making, hence they are also present in this release. Let’s detail the main ones.

Improved memory usage and speed for the --patch-from mode

V1.5.5 introduces memory-mapped dictionaries, by @daniellerozenblit, for both posix #3486 and windows #3557.

This feature allows zstd to memory-map large dictionaries, rather than requiring to load them into memory. This can make a pretty big difference for memory-constrained environments operating patches for large data sets. It's mostly visible under memory pressure, since mmap will be able to release less-used memory and continue working. But even when memory is plentiful, there are still measurable memory benefits, as shown in the graph below, especially when the reference turns out to be not completely relevant for the patch.

mmap_memory_usage

This feature is automatically enabled for --patch-from compression/decompression when the dictionary is larger than the user-set memory limit. It can also be manually enabled/disabled using --mmap-dict or --no-mmap-dict respectively.

Additionally, @daniellerozenblit introduces significant speed improvements for --patch-from.

An I/O optimization in #3486 greatly improves --patch-from decompression speed on Linux, typically by +50% on large files (~1GB).

patch-from_IO_optimization

Compression speed is also taken care of, with a dictionary-indexing speed optimization introduced in #3545. It wildly accelerates --patch-from compression, typically doubling speed on large files (~1GB), sometimes even more depending on exact scenario.

patch_from_compression_speed_optimization

This speed improvement comes at a slight regression in compression ratio, and is therefore enabled only on non-ultra compression strategies.

Speed improvements of middle-level compression for specific scenarios

The row-hash match finder introduced in version 1.5.0 for levels 5-12 has been improved in version 1.5.5, enhancing its speed in specific corner-case scenarios.

The first optimization (#3426) accelerates streaming compression using ZSTD_compressStream on small inputs by removing an expensive table initialization step. This results in remarkable speed increases for very small inputs.

The following scenario measures compression speed of ZSTD_compressStream at level 9 for different sample sizes on a linux platform running an i7-9700k cpu.

sample size v1.5.4 (MB/s) v1.5.5 (MB/s) improvement
100 1.4 44.8 x32
200 2.8 44.9 x16
500 6.5 60.0 x9.2
1K 12.4 70.0 x5.6
2K 25.0 111.3 x4.4
4K 44.4 139.4 x3.2
... ... ...
1M 97.5 99.4 +2%

The second optimization (#3552) speeds up compression of incompressible data by a large multiplier. This is achieved by increasing the step size and reducing the frequency of matching when no matches are found, with negligible impact on the compression ratio. It makes mid-level compression essentially inexpensive when processing incompressible data, typically, already compressed data (note: this was already the case for fast compression levels).

The following scenario measures compression speed of ZSTD_compress compiled with gcc-9 for a ~10MB incompressible sample on a linux platform running an i7-9700k cpu.

level v1.5.4 (MB/s) v1.5.5 (MB/s) improvement
3 3500 3500 not a row-hash level (control)
5 400 2500 x6.2
7 380 2200 x5.8
9 176 1880 x10
11 67 1130 x16
13 89 89 not a row-hash level (control)

Miscellaneous

There are other welcome speed improvements in this package.

For example, @felixhandte managed to increase processing speed of small files by carefully reducing the nb of system calls (#3479). This can easily translate into +10% speed when processing a lot of small files in batch.

The Seekable format received a bit of care. It's now much faster when splitting data into very small blocks (#3544). In an extreme scenario reported by @P-E-Meunier, it improves processing speed by x90. Even for more "common" settings, such as using 4KB blocks on some "normally" compressible data like enwik, it still provides a healthy x2 processing speed benefit. Moreover, @dloidolt merged an optimization that reduces the nb of I/O seek() events during reads (decompression), which is also beneficial for speed.

The release is not limited to speed improvements, several loose ends and corner cases were also fixed in this release. For a more detailed list of changes, please take a look at the changelog.

Change Log

Full change list (auto-generated)

New Contributors

Full Changelog: https://github.com/facebook/zstd/compare/v1.5.4...v1.5.5

相关地址：原始地址 下载(tar) 下载(zip)

1、 zstd-1.5.5.tar.gz 2.26MB

2、 zstd-1.5.5.tar.gz.sha256 84B

3、 zstd-1.5.5.tar.gz.sig 858B

4、 zstd-1.5.5.tar.zst 1.73MB

5、 zstd-1.5.5.tar.zst.sha256 85B

6、 zstd-1.5.5.tar.zst.sig 858B

7、 zstd-v1.5.5-win32.zip 1.57MB

8、 zstd-v1.5.5-win64.zip 1.61MB

查看：2023-04-05发行的版本

微信小程序
MyGit:GitHub仓库更新&通知小工具