librdkafka v1.2.0 release

WARNING: There is an issue with SASL GSSAPI authentication on Windows with this release. Upgrade directly to v1.2.1 which fixes the issue.

v1.2.0 is a feature release making the consumer transaction aware.

• Transaction aware consumer (isolation.level=read_committed) implemented by @mhowlett.
• Sub-millisecond buffering (linger.ms) on the producer.
• Improved authentication errors (KIP-152)

Consumer-side transaction support

This release adds consumer-side support for transactions. In previous releases, the consumer always delivered all messages to the application, even those in aborted or not yet committed transactions. In this release, the consumer will by default skip messages in aborted transactions. This is controlled through the new isolation.level configuration property which defaults to read_committed (only read committed messages, filter out aborted and not-yet committed transactions), to consume all messages, including for aborted transactions, you may set this property to read_uncommitted to get the behaviour of previous releases. For consumers in read_committed mode, the end of a partition is now defined to be the offset of the last message of a successfully committed transaction (referred to as the 'Last Stable Offset'). For non-transactional messages there is no change from previous releases, they will always be read, but a consumer will not advance into a not yet committed transaction on the partition.

Upgrade considerations

• linger.ms default was changed from 0 to 0.5 ms to promote some level of batching even with default settings.

New configuration properties

• Consumer property isolation.level=read_committed ensures the consumer will only read messages from successfully committed producer transactions. Default is read_committed. To get the previous behaviour, set the property to read_uncommitted, which will read all messages produced to a topic, regardless if the message was part of an aborted or not yet committed transaction.

Enhancements

• Offset commit metadata (arbitrary application-specified data) is now returned by rd_kafka_committed() and rd_kafka_offsets_for_times() (@damour, #2393)
• C++: Added Conf::c_ptr*() to retrieve the underlying C config object.
• Added on_thread_start() and on_thread_exit() interceptors.
• Increase queue.buffering.max.kbytes max to INT_MAX.
• Optimize varint decoding, increasing consume performance by ~15%.

Fixes

General:

• Rate limit IO-based queue wakeups to linger.ms, this reduces CPU load and lock contention for high throughput producer applications. (#2509)
• Reduce memory allocations done by rd_kafka_topic_partition_list_new().
• Fix socket recv error handling on MSVC (by Jinsu Lee).
• Avoid 1s stalls in some scenarios when broker wakeup-fd is triggered.
• SSL: Use only hostname (not port) when valid broker hostname (by Hunter Jacksson)
• SSL: Ignore OpenSSL cert verification results if enable.ssl.certificate.verification=false (@salisbury-espinosa, #2433)
• rdkafka_example_cpp: fix metadata listing mode (@njzcx)
• SASL Kerberos/GSSAPI: don't treat kinit ECHILD errors as errors (@hannip, #2421)
• Fix compare overflows (#2443)
• configure: Add option to disable automagic dependency on zstd (by Thomas Deutschmann)
• Documentation updates and fixes by Cedric Cellier and @ngrandem
• Set thread name on MacOS X (by Nikhil Benesch)
• C++: Fix memory leak in Headers (by Vladimir Sakharuk)
• Fix UBSan (undefined behaviour errors) (@PlacidBox, #2417)
• CONFIGURATION.md: escape || inside markdown table (@mhowlett)
• Refresh broker list metadata even if no topics to refresh (#2476)

Consumer:

• Make rd_kafka_pause|resume_partitions() synchronous, making sure that a subsequent consumer_poll() will not return messages for the paused partitions (#2455).
• Fix incorrect toppar destroy in OffsetRequest (@binary85, #2379)
• Fix message version 1 offset calculation (by Martin Ivanov)
• Defer commit in transport error to avoid consumer_close hang.

Producer:

• Messages were not timed out for leader-less partitions (.NET issue #1027).
• Improve message timeout granularity to millisecond precision (the smallest ffective message timeout will still be 1000ms).
• message.timeout.ms=0 is now accepted even if linger.ms > 0 (by Jeff Snyder)
• Don't track max.poll.interval.ms unless in Consumer mode, this saves quite a few memory barries for high-performance Producers.
• Optimization: avoid atomic fatal error code check when idempotence is disabled.

Checksums

Release asset checksums:

• v1.2.0.zip SHA256 6e57f09c28e9a65abb886b84ff638b2562b8ad71572de15cf58578f3f9bc45ec
• v1.2.0.tar.gz SHA256 eedde1c96104e4ac2d22a4230e34f35dd60d53976ae2563e3dd7c27190a96859