Zstd v1.4.5 Release Notes

This is a fairly important release which includes performance improvements and new major CLI features. It also fixes a few corner cases, making it a recommended upgrade.

Faster Decompression Speed

Decompression speed has been improved again, thanks to great contributions from @terrelln. As usual, exact mileage varies depending on files and compilers. For x64 cpus, expect a speed bump of at least +5%, and up to +10% in favorable cases. ARM cpus receive more benefit, with speed improvements ranging from +15% vicinity, and up to +50% for certain SoCs and scenarios (ARM‘s situation is more complex due to larger differences in SoC designs).

For illustration, some benchmarks run on a modern x64 platform using zstd -b compiled with gcc v9.3.0 :

Same platform, using clang v10.0.0 compiler :

Simplified integration

Presuming a project needs to integrate libzstd's source code (as opposed to linking a pre-compiled library), the /lib source directory can be copy/pasted into target project. Then the local build system must setup a few include directories. Some setups are automatically provided in prepared build scripts, such as Makefile, but any other 3rd party build system must do it on its own. This integration is now simplified, thanks to @felixhandte, by making all dependencies within /lib relative, meaning it’s only necessary to setup include directories for the *.h header files that are directly included into target project (typically zstd.h). Even that task can be circumvented by copy/pasting the *.h into already established include directories.

Alternatively, if you are a fan of one-file integration strategy, @cwoffenden has extended his one-file decoder script into a full feature one-file compression library. The script create_single_file_library.sh will generate a file zstd.c, which contains all selected elements from the library (by default, compression and decompression). It’s then enough to import just zstd.h and the generated zstd.c into target project to access all included capabilities.

--patch-from

Zstandard CLI is introducing a new command line option --patch-from, which leverages existing compressors, dictionaries and long range match finder to deliver a high speed engine for producing and applying patches to files.

--patch-from is based on dictionary compression. It will consider a previous version of a file as a dictionary, to better compress a new version of same file. This operation preserves fast zstd speeds at lower compression levels. To this ends, it also increases the previous maximum limit for dictionaries from 32 MB to 2 GB, and automatically uses the long range match finder when needed (though it can also be manually overruled). --patch-from can also be combined with multi-threading mode at a very minimal compression ratio loss.

Example usage:

# create the patch
zstd --patch-from=<oldfile> <newfile> -o <patchfile>

# apply the patch
zstd -d --patch-from=<oldfile> <patchfile> -o <newfile>`

Benchmarks: We compared zstd to bsdiff, a popular industry grade diff engine. Our test corpus were tarballs of different versions of source code from popular GitHub repositories. Specifically:

`repos = {
    # ~31mb (small file)
    "zstd": {"url": "https://github.com/facebook/zstd", "dict-branch": "refs/tags/v1.4.2", "src-branch": "refs/tags/v1.4.3"},
    # ~273mb (medium file)
    "wordpress": {"url": "https://github.com/WordPress/WordPress", "dict-branch": "refs/tags/5.3.1", "src-branch": "refs/tags/5.3.2"},
    # ~1.66gb (large file)
    "llvm": {"url": "https://github.com/llvm/llvm-project", "dict-branch": "refs/tags/llvmorg-9.0.0", "src-branch": "refs/tags/llvmorg-9.0.1"}
}`

--patch-from on level 19 (with chainLog=30 and targetLength=4kb) is comparable with bsdiff when comparing patch sizes.

--patch-from greatly outperforms bsdiff in speed even on its slowest setting of level 19 boasting an average speedup of ~7X. --patch-from is >200X faster on level 1 and >100X faster (shown below) on level 3 vs bsdiff while still delivering patch sizes less than 0.5% of the original file size.

And of course, there is no change to the fast zstd decompression speed.

Addendum :

After releasing --patch-from, we were made aware of two other popular diff engines by the community: SmartVersion and Xdelta. We ran some additional benchmarks for them and here are our primary takeaways. All three tools are excellent diff engines with clear advantages (especially in speed) over the popular bsdiff. Patch sizes for both binary and text data produced by all three are pretty comparable with Xdelta underperforming Zstd and SmartVersion only slightly [1]. For patch creation speed, Xdelta is the clear winner for text data and Zstd is the clear winner for binary data [2]. And for Patch Extraction Speed (ie. decompression), Zstd is fastest in all scenarios [3]. See wiki for details.

--filelist=

Finally, --filelist= is a new CLI capability, which makes it possible to pass a list of files to operate upon from a file, as opposed to listing all target files solely on the command line. This makes it possible to prepare a list offline, save it into a file, and then provide the prepared list to zstd. Another advantage is that this method circumvents command line size limitations, which can become a problem when operating on very large directories (such situation can typically happen with shell expansion). In contrast, passing a very large list of filenames from within a file is free of such size limitation.

Full List

• perf: Improved decompression speed (x64 >+5%, ARM >+15%), by @terrelln
• perf: Automatically downsizes ZSTD_DCtx when too large for too long (#2069, by @bimbashrestha)
• perf: Improved fast compression speed on aarch64 (#2040, ~+3%, by @caoyzh)
• perf: Small level 1 compression speed gains (depending on compiler)
• fix: Compression ratio regression on huge files (> 3 GB) using high levels (--ultra) and multithreading, by @terrelln
• api: ZDICT_finalizeDictionary() is promoted to stable (#2111)
• api: new experimental parameter ZSTD_d_stableOutBuffer (#2094)
• build: Generate a single-file libzstd library (#2065, by @cwoffenden)
• build: Relative includes, no longer require -I flags for zstd lib subdirs (#2103, by @felixhandte)
• build: zstd now compiles cleanly under -pedantic (#2099)
• build: zstd now compiles with make-4.3
• build: Support mingw cross-compilation from Linux, by @Ericson2314
• build: Meson multi-thread build fix on windows
• build: Some misc icc fixes backed by new ci test on travis
• cli: New --patch-from command, create and apply patches from files, by @bimbashrestha
• cli: --filelist= : Provide a list of files to operate upon from a file
• cli: -b can now benchmark multiple files in decompression mode
• cli: New --no-content-size command
• cli: New --show-default-cparams command
• misc: new diagnosis tool, checked_flipped_bits, in contrib/, by @felixhandte
• misc: Extend largeNbDicts benchmark to compression
• misc: experimental edit-distance match finder in contrib/
• doc: Improved beginner CONTRIBUTING.md docs
• doc: New issue templates for zstd