librdkafka v2.0.0 is a feature release:

• KIP-88 OffsetFetch Protocol Update (#3995).
• KIP-222 Add Consumer Group operations to Admin API (started by @lesterfan, #3995).
• KIP-518 Allow listing consumer groups per state (#3995).
• KIP-396 Partially implemented: support for AlterConsumerGroupOffsets (started by @lesterfan, #3995).
• OpenSSL 3.0.x support - the maximum bundled OpenSSL version is now 3.0.7 (previously 1.1.1q).
• Fixes to the transactional and idempotent producer.

Upgrade considerations

OpenSSL 3.0.x

OpenSSL default ciphers

The introduction of OpenSSL 3.0.x in the self-contained librdkafka bundles changes the default set of available ciphers, in particular all obsolete or insecure ciphers and algorithms as listed in the OpenSSL legacy manual page are now disabled by default.

WARNING: These ciphers are disabled for security reasons and it is highly recommended NOT to use them.

Should you need to use any of these old ciphers you'll need to explicitly enable the legacy provider by configuring ssl.providers=default,legacy on the librdkafka client.

OpenSSL engines and providers

OpenSSL 3.0.x deprecates the use of engines, which is being replaced by providers. As such librdkafka will emit a deprecation warning if ssl.engine.location is configured.

OpenSSL providers may be configured with the new ssl.providers configuration property.

Broker TLS certificate hostname verification

The default value for ssl.endpoint.identification.algorithm has been changed from none (no hostname verification) to https, which enables broker hostname verification (to counter man-in-the-middle impersonation attacks) by default.

To restore the previous behaviour, set ssl.endpoint.identification.algorithm to none.

Known Issues

Poor Consumer batch API messaging guarantees

The Consumer Batch APIs rd_kafka_consume_batch() and rd_kafka_consume_batch_queue() are not thread safe if rkmessages_size is greater than 1 and any of the seek, pause, resume or rebalancing operation is performed in parallel with any of the above APIs. Some of the messages might be lost, or erroneously returned to the application, in the above scenario.

It is strongly recommended to use the Consumer Batch APIs and the mentioned operations in sequential order in order to get consistent result.

For rebalancing operation to work in sequencial manner, please set rebalance_cb configuration property (refer examples/rdkafka_complex_consumer_example.c for the help with the usage) for the consumer.

Enhancements

• Self-contained static libraries can now be built on Linux arm64 (#4005).
• Updated to zlib 1.2.13, zstd 1.5.2, and curl 7.86.0 in self-contained librdkafka bundles.
• Added on_broker_state_change() interceptor
• The C++ API no longer returns strings by const value, which enables better move optimization in callers.
• Added rd_kafka_sasl_set_credentials() API to update SASL credentials.
• Setting allow.auto.create.topics will no longer give a warning if used by a producer, since that is an expected use case. Improvement in documentation for this property.
• Added a resolve_cb configuration setting that permits using custom DNS resolution logic.
• Added rd_kafka_mock_broker_error_stack_cnt().
• The librdkafka.redist NuGet package has been updated to have fewer external dependencies for its bundled librdkafka builds, as everything but cyrus-sasl is now built-in. There are bundled builds with and without linking to cyrus-sasl for maximum compatibility.
• Admin API DescribeGroups() now provides the group instance id for static members KIP-345 (#3995).

Fixes

General fixes

• Windows: couldn't read a PKCS#12 keystore correctly because binary mode wasn't explicitly set and Windows defaults to text mode.
• Fixed memory leak when loading SSL certificates (@Mekk, #3930)
• Load all CA certificates from ssl.ca.pem, not just the first one.
• Each HTTP request made when using OAUTHBEARER OIDC would leak a small amount of memory.

Transactional producer fixes

• When a PID epoch bump is requested and the producer is waiting to reconnect to the transaction coordinator, a failure in a find coordinator request could cause an assert to fail. This is fixed by retrying when the coordinator is known (#4020).
• Transactional APIs (except send_offsets_for_transaction()) that timeout due to low timeout_ms may now be resumed by calling the same API again, as the operation continues in the background.
• For fatal idempotent producer errors that may be recovered by bumping the epoch the current transaction must first be aborted prior to the epoch bump. This is now handled correctly, which fixes issues seen with fenced transactional producers on fatal idempotency errors.
• Timeouts for EndTxn requests (transaction commits and aborts) are now automatically retried and the error raised to the application is also a retriable error.
• TxnOffsetCommitRequests were retried immediately upon temporary errors in send_offsets_to_transactions(), causing excessive network requests. These retries are now delayed 500ms.
• If init_transactions() is called with an infinite timeout (-1), the timeout will be limited to 2 * transaction.timeout.ms. The application may retry and resume the call if a retriable error is returned.

Consumer fixes

• Back-off and retry JoinGroup request if coordinator load is in progress.
• Fix rd_kafka_consume_batch() and rd_kafka_consume_batch_queue() skipping other partitions' offsets intermittently when seek, pause, resume or rebalancing is used for a partition.
• Fix rd_kafka_consume_batch() and rd_kafka_consume_batch_queue() intermittently returing incorrect partitions' messages if rebalancing happens during these operations.

Checksums

Release asset checksums:

• v2.0.0.zip SHA256 9d8a8be30ed09daf6c560f402e91db22fcaea11cac18a0d3c0afdbf884df1d4e
• v2.0.0.tar.gz SHA256 f75de3545b3c6cc027306e2df0371aefe1bb8f86d4ec612ed4ebf7bfb2f817cd