Zstandard v1.3.5 is a maintenance release focused on dictionary compression performance.

Compression is generally associated with the act of willingly requesting the compression of some large source. However, within datacenters, compression brings its best benefits when completed transparently. In such scenario, it's actually very common to compress a large number of very small blobs (individual messages in a stream or log, or records in a cache or datastore, etc.). Dictionary compression is a great tool for these use cases.

This release makes dictionary compression significantly faster for these situations, when compressing small to very small data (inputs up to ~16 KB).

The above image plots the compression speeds at different input sizes for zstd v1.3.4 (red) and v1.3.5 (green), at levels 1, 3, 9, and 18. The benchmark data was gathered on an Intel Xeon CPU E5-2680 v4 @ 2.40GHz. The benchmark was compiled with clang-7.0, with the flags -O3 -march=native -mtune=native -DNDEBUG. The file used in the results shown here is the osdb file from the Silesia corpus, cut into small blocks. It was selected because it performed roughly in the middle of the pack among the Silesia files.

The new version saves substantial initialization time, which is increasingly important as the average size to compress becomes smaller. The impact is even more perceptible at higher levels, where initialization costs are higher. For larger inputs, performance remain similar.

Users can expect to measure substantial speed improvements for inputs smaller than 8 KB, and up to 32 KB depending on the context. The expected speed-up ranges from none (large, incompressible blobs) to many times faster (small, highly compressible inputs). Real world examples up to 15x have been observed.

Other noticeable improvements

The compression levels have been slightly adjusted, taking into consideration the higher top speed of level 1 since v1.3.4, and making level 19 a substantially stronger compression level while preserving the 8 MB window size limit, hence keeping an acceptable memory budget for decompression.

It's also possible to select the content of libzstd by modifying macro values at compilation time. By default, libzstd contains everything, but its size can be made substantially smaller by removing support for the dictionary builder, or legacy formats, or deprecated functions. It's even possible to build a compression-only or a decompression-only library.

Detailed changes list

• perf: much faster dictionary compression, by @felixhandte
• perf: small quality improvement for dictionary generation, by @terrelln
• perf: improved high compression levels (notably level 19)
• mem : automatic memory release for long duration contexts
• cli : fix : overlapLog can be manually set
• cli : fix : decoding invalid lz4 frames
• api : fix : performance degradation for dictionary compression when using advanced API, by @terrelln
• api : change : clarify ZSTD_CCtx_reset() vs ZSTD_CCtx_resetParameters(), by @terrelln
• build: select custom libzstd scope through control macros, by @GeorgeLu97
• build: OpenBSD support, by @bket
• build: make and make all are compatible with -j
• doc : clarify zstd_compression_format.md, updated for IETF RFC process
• misc: pzstd compatible with reproducible compilation, by @lamby

Known bug

zstd --list does not work with non-interactive tty. This issue is fixed in dev branch.