Advanced API

The main focus of the v1.4.0 release is the stabilization of the advanced API.

The advanced API provides a way to set specific parameters during compression and decompression in an API and ABI compatible way. For example, it allows you to compress with multiple threads, enable --long mode, set frame parameters, and load dictionaries. It is compatible with ZSTD_compressStream*() and ZSTD_compress2(). There is also an advanced decompression API that allows you to set parameters like maximum memory usage, and load dictionaries. It is compatible with the existing decompression functions ZSTD_decompressStream() and ZSTD_decompressDCtx().

The old streaming functions are all compatible with the new API, and the documentation provides the equivalent function calls in the new API. For example, see ZSTD_initCStream(). The stable functions will remain supported, but the functions in the experimental sections, like ZSTD_initCStream_usingDict(), will eventually be marked as deprecated and removed in favor of the new advanced API.

The examples have all been updated to use the new advanced API. If you have questions about how to use the new API, please refer to the examples, and if they are unanswered, please open an issue.

Performance

Zstd's fastest compression level just got faster! Thanks to ideas from Intel's igzip and @gbtucker, we've made level 1, zstd's fastest strategy, 6-8% faster in most scenarios. For example on the Silesia Corpus with level 1, we see 0.2% better compression compared to zstd-1.3.8, and these performance figures on an Intel i9-9900K:

New Features

A new experimental function ZSTD_decompressBound() has been added by @shakeelrao. It is useful when decompressing zstd data in a single shot that may, or may not have the decompressed size written into the frame. It is exact when the decompressed size is written into the frame, and a tight upper bound within 128 KB, as long as ZSTD_e_flush and ZSTD_flushStream() aren't used. When ZSTD_e_flush is used, in the worst case the bound can be very large, but this isn't a common scenario.

The parameter ZSTD_c_literalCompressionMode and the CLI flag --[no-]compress-literals allow users to explicitly enable and disable literal compression. By default literals are compressed with positive compression levels, and left uncompressed for negative compression levels. Disabling literal compression boosts compression and decompression speed, at the cost of compression ratio.

Detailed list of changes

• perf: Improve level 1 compression speed in most scenarios by 6% by @gbtucker and @terrelln
• api: Move the advanced API, including all functions in the staging section, to the stable section
• api: Make ZSTD_e_flush and ZSTD_e_end block for maximum forward progress
• api: Rename ZSTD_CCtxParam_getParameter to ZSTD_CCtxParams_getParameter
• api: Rename ZSTD_CCtxParam_setParameter to ZSTD_CCtxParams_setParameter
• api: Don't export ZSTDMT functions from the shared library by default
• api: Require ZSTD_MULTITHREAD to be defined to use ZSTDMT
• api: Add ZSTD_decompressBound() to provide an upper bound on decompressed size by @shakeelrao
• api: Fix ZSTD_decompressDCtx() corner cases with a dictionary
• api: Move ZSTD_getDictID_*() functions to the stable section
• api: Add ZSTD_c_literalCompressionMode flag to enable or disable literal compression by @terrelln
• api: Allow compression parameters to be set when a dictionary is used
• api: Allow setting parameters before or after ZSTD_CCtx_loadDictionary() is called
• api: Fix ZSTD_estimateCStreamSize_usingCCtxParams()
• api: Setting ZSTD_d_maxWindowLog to 0 means use the default
• cli: Ensure that a dictionary is not used to compress itself by @shakeelrao
• cli: Add --[no-]compress-literals flag to enable or disable literal compression
• doc: Update the examples to use the advanced API
• doc: Explain how to transition from old streaming functions to the advanced API in the header
• build: Improve the Windows release packages
• build: Improve CMake build by @hjmjohnson
• build: Build fixes for FreeBSD by @lwhsu
• build: Remove redundant warnings by @thatsafunnyname
• build: Fix tests on OpenBSD by @bket
• build: Extend fuzzer build system to work with the new clang engine
• build: CMake now creates the libzstd.so.1 symlink
• build: Improve Menson build by @lzutao
• misc: Fix symbolic link detection on FreeBSD
• misc: Use physical core count for -T0 on FreeBSD by @cemeyer
• misc: Fix zstd --list on truncated files by @kostmo
• misc: Improve logging in debug mode by @felixhandte
• misc: Add CirrusCI tests by @lwhsu
• misc: Optimize dictionary memory usage in corner cases
• misc: Improve the dictionary builder on small or homogeneous data
• misc: Fix spelling across the repo by @jsoref