librdkafka v1.6.0

librdkafka v1.6.0 is feature release:

• KIP-429 Incremental rebalancing with sticky consumer group partition assignor (KIP-54) (by @mhowlett).
• KIP-480 Sticky producer partitioning (sticky.partitioning.linger.ms) - achieves higher throughput and lower latency through sticky selection of random partition (by @abbycriswell).
• AdminAPI: Add support for DeleteRecords(), DeleteGroups() and DeleteConsumerGroupOffsets() (by @gridaphobe)
• KIP-447 Producer scalability for exactly once semantics - allows a single transactional producer to be used for multiple input partitions. Requires Apache Kafka 2.5 or later.
• Transactional producer fixes and improvements, see Transactional Producer fixes below.
• The librdkafka.redist NuGet package now supports Linux ARM64/Aarch64.

Upgrade considerations

• Sticky producer partitioning (sticky.partitioning.linger.ms) is enabled by default (10 milliseconds) which affects the distribution of randomly partitioned messages, where previously these messages would be evenly distributed over the available partitions they are now partitioned to a single partition for the duration of the sticky time (10 milliseconds by default) before a new random sticky partition is selected.
• The new KIP-447 transactional producer scalability guarantees are only supported on Apache Kafka 2.5 or later, on earlier releases you will need to use one producer per input partition for EOS. This limitation is not enforced by the producer or broker.
• Error handling for the transactional producer has been improved, see the Transactional Producer fixes below for more information.

Known issues

• The Transactional Producer's API timeout handling is inconsistent with the underlying protocol requests, it is therefore strongly recommended that applications call rd_kafka_commit_transaction() and rd_kafka_abort_transaction() with the timeout_ms parameter set to -1, which will use the remaining transaction timeout.

Enhancements

• KIP-107, KIP-204: AdminAPI: Added DeleteRecords() (by @gridaphobe).
• KIP-229: AdminAPI: Added DeleteGroups() (by @gridaphobe).
• KIP-496: AdminAPI: Added DeleteConsumerGroupOffsets().
• KIP-464: AdminAPI: Added support for broker-side default partition count and replication factor for CreateTopics().
• Windows: Added ssl.ca.certificate.stores to specify a list of Windows Certificate Stores to read CA certificates from, e.g., CA,Root. Root remains the default store.
• Use reentrant rand_r() on supporting platforms which decreases lock contention (@azat).
• Added assignor debug context for troubleshooting consumer partition assignments.
• Updated to OpenSSL v1.1.1i when building dependencies.
• Update bundled lz4 (used when ./configure --disable-lz4-ext) to v1.9.3 which has vast performance improvements.
• Added rd_kafka_conf_get_default_topic_conf() to retrieve the default topic configuration object from a global configuration object.
• Added conf debugging context to debug - shows set configuration properties on client and topic instantiation. Sensitive properties are redacted.
• Added rd_kafka_queue_yield() to cancel a blocking queue call.
• Will now log a warning when multiple ClusterIds are seen, which is an indication that the client might be erroneously configured to connect to multiple clusters which is not supported.
• Added rd_kafka_seek_partitions() to seek multiple partitions to per-partition specific offsets.

Fixes

General fixes

• Fix a use-after-free crash when certain coordinator requests were retried.
• The C++ oauthbearer_set_token() function would call free() on a new-created pointer, possibly leading to crashes or heap corruption (#3194)

Consumer fixes

• The consumer assignment and consumer group implementations have been decoupled, simplified and made more strict and robust. This will sort out a number of edge cases for the consumer where the behaviour was previously undefined.
• Partition fetch state was not set to STOPPED if OffsetCommit failed.
• The session timeout is now enforced locally also when the coordinator connection is down, which was not previously the case.

Transactional Producer fixes

• Transaction commit or abort failures on the broker, such as when the producer was fenced by a newer instance, were not propagated to the application resulting in failed commits seeming successful. This was a critical race condition for applications that had a delay after producing messages (or sendings offsets) before committing or aborting the transaction. This issue has now been fixed and test coverage improved.
• The transactional producer API would return RD_KAFKA_RESP_ERR__STATE when API calls were attempted after the transaction had failed, we now try to return the error that caused the transaction to fail in the first place, such as RD_KAFKA_RESP_ERR__FENCED when the producer has been fenced, or RD_KAFKA_RESP_ERR__TIMED_OUT when the transaction has timed out.
• Transactional producer retry count for transactional control protocol requests has been increased from 3 to infinite, retriable errors are now automatically retried by the producer until success or the transaction timeout is exceeded. This fixes the case where rd_kafka_send_offsets_to_transaction() would fail the current transaction into an abortable state when CONCURRENT_TRANSACTIONS was returned by the broker (which is a transient error) and the 3 retries were exhausted.

Producer fixes

• Calling rd_kafka_topic_new() with a topic config object with message.timeout.ms set could sometimes adjust the global linger.ms property (if not explicitly configured) which was not desired, this is now fixed and the auto adjustment is only done based on the default_topic_conf at producer creation.
• rd_kafka_flush() could previously return RD_KAFKA_RESP_ERR__TIMED_OUT just as the timeout was reached if the messages had been flushed but there were now no more messages. This has been fixed.

Checksums

Release asset checksums:

• v1.6.0.zip SHA256 af6f301a1c35abb8ad2bb0bab0e8919957be26c03a9a10f833c8f97d6c405aa8
• v1.6.0.tar.gz SHA256 3130cbd391ef683dc9acf9f83fe82ff93b8730a1a34d0518e93c250929be9f6b