Advanced API

v1.3.8 main focus is the stabilization of the advanced API.

This API has been in the making for more than a year, and makes it possible to trigger advanced features, such as multithreading, --long mode, or detailed frame parameters, in a straightforward and extensible manner. Some examples are provided in this blog entry. To make this vision possible, the advanced API relies on sticky parameters, which can be stacked on top of each other in any order. This makes it possible to introduce new features in the future without breaking API nor ABI.

This API has provided a good experience in our infrastructure, and we hope it will prove easy to use and efficient in your applications. Nonetheless, before being branded "stable", this proposal must spend a last round in "staging area", in order to generate comments and feedback from new users. It's planned to be labelled "stable" by v1.4.0, which is expected to be next release, depending on received feedback.

The experimental section still contains a lot of prototypes which are largely redundant with the new advanced API. Expect them to become deprecated, and then later dropped in some future. Transition towards the newer advanced API is therefore highly recommended.

Performance

Decoding speed has been improved again, primarily for some specific scenarios : frames using large window sizes (--ultra or --long), and cold dictionary. Cold dictionary is expected to become more important in the near future, as solutions relying on thousands of dictionaries simultaneously will be deployed.

The higher compression levels get a slight compression ratio boost, mostly visible for small (<256 KB) and large (>32 MB) data streams. This change benefits asymmetric scenarios (compress ones, decompress many times), typically targeting level 19.

New features

A noticeable addition, @terrelln introduces the --rsyncable mode to zstd. Similar to gzip --rsyncable, it generates a compressed frame which is friendly to rsync in case of limited changes : a difference in the input data will only impact a small localized amount of compressed data, instead of everything from the position onward due to cascading impacts. This is useful for very large archives regularly updated and synchronized over long distance connections (as an example, compressed mailboxes come to mind).

The method used by zstd preserves the compression ratio very well, introducing only very tiny losses due to synchronization points, meaning it's no longer a sacrifice to use --rsyncable. Here is an example on silesia.tar, using default compression level :

Speaking of compression of level : it's now possible to use environment variable ZSTD_CLEVEL to influence default compression level. This can prove useful in situations where it's not possible to provide command line parameters, typically when zstd is invoked "under the hood" by some calling process.

Lastly, anyone interested in embedding a small zstd decoder into a space-constrained application will be interested in a new set of build macros introduced by @felixhandte, which makes it possible to selectively turn off decoder features to reduce binary size even further. Final binary size will of course vary depending on target assembler and compiler, but in preliminary testings on x64, it helped reducing the decoder size by a factor 3 (from ~64KB towards ~20KB).

Detailed list of changes

• perf: better decompression speed on large files (+7%) and cold dictionaries (+15%)
• perf: slightly better compression ratio at high compression modes
• api : finalized advanced API, last stage before "stable" status
• api : new --rsyncable mode, by @terrelln
• api : support decompression of empty frames into NULL (used to be an error) (#1385)
• build: new set of build macros to generate a minimal size decoder, by @felixhandte
• build: fix compilation on MIPS32, reported by @clbr (#1441)
• build: fix compilation with multiple -arch flags, by @ryandesign
• build: highly upgraded meson build, by @lzutao
• build: improved buck support, by @obelisk
• build: fix cmake script : can create debug build, by @pitrou
• build: Makefile : grep works on both colored consoles and systems without color support
• build: fixed zstd-pgo target, by @bmwiedemann
• cli : support ZSTD_CLEVEL environment variable, by @yijinfb (#1423)
• cli : --no-progress flag, preserving final summary (#1371), by @terrelln
• cli : ensure destination file is not source file (#1422)
• cli : clearer error messages, notably when input file not present
• doc : clarified zstd_compression_format.md, by @ulikunitz
• misc: fixed zstdgrep, returns 1 on failure, by @lzutao
• misc: NEWS renamed as CHANGELOG, in accordance with fb.oss policy