LZ4 v1.8.1 most visible new feature is its support for Dictionary compression . This was already somewhat possible, but in a complex way, requiring knowledge of internal working. Support is now more formally added on the API side within lib/lz4frame_static.h. It's early days, and this new API is tagged "experimental" for the time being.

Support is also added in the command line utility lz4, using the new command -D, implemented by @felixhandte. The behavior of this command is identical to zstd, should you be already familiar.

lz4 doesn't specify how to build a dictionary. All it says is that it can be any file up to 64 KB. This approach is compatible with zstd dictionary builder, which can be instructed to create a 64 KB dictionary with this command :

zstd --train dirSamples/* -o dictName --maxdict=64KB

LZ4 v1.8.1 also offers improved performance at ultra settings (levels 10+). These levels receive a new code, called optimal parser, available in lib/lz4_opt.h. Compared with previous version, the new parser uses less memory (from 384KB to 256KB), performs faster, compresses a little bit better (not much, as it was already close to theoretical limit), and resists pathological patterns which could destroy performance (see #339),

For comparison, here are some quick benchmark using LZ4 v1.8.0 on my laptop with silesia.tar :

./lz4 -b9e12 -v ~/dev/bench/silesia.tar
*** LZ4 command line interface 64-bits v1.8.0, by Yann Collet ***
Benchmarking levels from 9 to 12
 9#silesia.tar       : 211984896 ->  77897777 (2.721),  24.2 MB/s ,2401.8 MB/s
10#silesia.tar       : 211984896 ->  77852187 (2.723),  16.9 MB/s ,2413.7 MB/s
11#silesia.tar       : 211984896 ->  77435086 (2.738),   7.1 MB/s ,2425.7 MB/s
12#silesia.tar       : 211984896 ->  77274453 (2.743),   3.3 MB/s ,2390.0 MB/s

and now using LZ4 v1.8.1 :

./lz4 -b9e12 -v ~/dev/bench/silesia.tar
*** LZ4 command line interface 64-bits v1.8.1, by Yann Collet ***
Benchmarking levels from 9 to 12
 9#silesia.tar       : 211984896 ->  77890594 (2.722),  24.4 MB/s ,2405.2 MB/s
10#silesia.tar       : 211984896 ->  77859538 (2.723),  19.3 MB/s ,2476.0 MB/s
11#silesia.tar       : 211984896 ->  77369725 (2.740),  10.1 MB/s ,2478.4 MB/s
12#silesia.tar       : 211984896 ->  77270146 (2.743),   3.7 MB/s ,2508.3 MB/s

The new parser is also directly compatible with lower compression levels, which brings additional benefits :

• Compatibility with LZ4_*_destSize() variant, which reverses the logic by trying to fit as much data as possible into a predefined limited size buffer.
• Compatibility with Dictionary compression, as it uses the same tables as regular HC mode

In the future, this compatibility will also allow dynamic on-the-fly change of compression level, but such feature is not implemented at this stage.

The release also provides a set of small bug fixes and improvements, listed below :

• perf : faster and stronger ultra modes (levels 10+)
• perf : slightly faster compression and decompression speed
• perf : fix bad degenerative case, reported by @c-morgenstern
• fix : decompression failed when using a combination of extDict + low memory address (#397), reported and fixed by Julian Scheid (@jscheid)
• cli : support for dictionary compression (-D), by Felix Handte @felixhandte
• cli : fix : lz4 -d --rm preserves timestamp (#441)
• cli : fix : do not modify /dev/null permission as root, by @aliceatlas
• api : new dictionary api in lib/lz4frame_static.h
• api : _destSize() variant supported for all compression levels
• build : make and make test compatible with parallel build -jX, reported by @mwgamera
• build : can control LZ4LIB_VISIBILITY macro, by @mikir
• install: fix man page directory (#387), reported by Stuart Cardall (@itoffshore)

Note : v1.8.1.2 is the same as v.1.8.1, with the version number fixed in source code, as notified by Po-Chuan Hsieh (@sunpoet).