librdkafka v1.4.0

v1.4.0 is a feature release:

• KIP-98: Transactional Producer API
• KIP-345: Static consumer group membership (by @rnpridgeon)
• KIP-511: Report client software name and version to broker

Transactional Producer API

librdkafka now has complete Exactly-Once-Semantics (EOS) functionality, supporting the idempotent producer (since v1.0.0), a transaction-aware consumer (since v1.2.0) and full producer transaction support (in this release). This enables developers to create Exactly-Once applications with Apache Kafka.

See the Transactions in Apache Kafka page for an introduction and check the librdkafka transactions example for a complete transactional application example.

Security fixes

Two security issues have been identified in the SASL SCRAM protocol handler:

• The client nonce, which is expected to be a random string, was a static string.
• If sasl.username and sasl.password contained characters that needed escaping, a buffer overflow and heap corruption would occur. This was protected, but too late, by an assertion.

Both of these issues are fixed in this release.

Enhancements

• Add FNV-1a partitioner (by @Manicben, #2724). The new fnv1a_random partitioner is compatible with Sarama's NewHashPartitioner partition, easing transition from Sarama to librdkafka-based clients such as confluent-kafka-go.
• Added rd_kafka_error_t / RdKafka::Error complex error type which provides error attributes such as indicating if an error is retriable.
• The builtin mock broker now supports balanced consumer groups.
• Support finding headers in nonstandard directories in CMake (@benesch)
• Improved static library bundles which can now contain most dependencies.
• Documentation, licenses, etc, is now installed by make install
• Bump OpenSSL to v1.0.2u (when auto-building dependencies)

Fixes

General:

• Correct statistics names in docs (@TimWSpence, #2754)
• Wake up broker thread based on next request retry. Prior to this fix the next wakeup could be delayed up to 1 second regardless of next retry.
• Treat SSL peer resets as usual Disconnects, making log.connection.close work
• Reset buffer corrid on connection close to honour ApiVers and Sasl request priorities (@xzxxzx401, #2666)
• Cleanup conf object if failing to creat producer or consumer (@fboranek)
• Fix build of rdkafka_example project for windows, when using building it using Visual Studio 2017/2019 (by @Eliyahu-Machluf)
• Minor fix to rdkafka_example usage: add lz4 and zstd compression codec to usage (by @Eliyahu-Machluf)
• Let broker nodename updates propagate as ERR__TRANSPORT rather than ERR__NODE_UPDATE to avoid an extra error code for the application to handle.
• Fix erroneous refcount assert in enq_once_del_source (e.g., on admin operation timeout)
• Producers could get stuck in INIT state after a disconnect until a to-be-retried request timed out or the connection was needed for other purposes (metadata discovery, etc), this is now fixed.

Producer:

• flush() now works with RD_KAFKA_EVENT_DR
• Fix race condition when finding EOS-supporting broker

Consumer:

• Consumers could get stuck after rebalance if assignment was empty
• Enforce session.timeout.ms in the consumer itself (#2631)
• max.poll.interval.ms is now only enforced when using subscribe()
• Fix consumer_lag calculation for transactional topics
• Show fetch/no-fetch reason in topic debugging
• Properly propagate commit errors per partition
• Don't send heartbeats after max.poll.interval.ms is exceeded.
• Honour array size in rd_kafka_event_message_array() to avoid overflow (#2773)

Checksums

Release asset checksums:

• v1.4.0.zip SHA256 eaf954e3b8a2ed98360b2c76f55048ee911964de8aefd8a9e1133418ec9f48dd
• v1.4.0.tar.gz SHA256 ae27ea3f3d0d32d29004e7f709efbba2666c5383a107cc45b3a1949486b2eb84