librdkafka v2.3.0 is a feature release:

• KIP-516 Partial support of topic identifiers. Topic identifiers in metadata response available through the new rd_kafka_DescribeTopics function (#4300, #4451).
• KIP-117 Add support for AdminAPI DescribeCluster() and DescribeTopics() (#4240, @jainruchir).
• KIP-430: Return authorized operations in Describe Responses. (#4240, @jainruchir).
• KIP-580: Added Exponential Backoff mechanism for retriable requests with retry.backoff.ms as minimum backoff and retry.backoff.max.ms as the maximum backoff, with 20% jitter (#4422).
• KIP-396: completed the implementation with the addition of ListOffsets (#4225).
• Fixed ListConsumerGroupOffsets not fetching offsets for all the topics in a group with Apache Kafka version below 2.4.0.
• Add missing destroy that leads to leaking partition structure memory when there are partition leader changes and a stale leader epoch is received (#4429).
• Fix a segmentation fault when closing a consumer using the cooperative-sticky assignor before the first assignment (#4381).
• Fix for insufficient buffer allocation when allocating rack information (@wolfchimneyrock, #4449).
• Fix for infinite loop of OffsetForLeaderEpoch requests on quick leader changes. (#4433).
• Fix to add leader epoch to control messages, to make sure they're stored for committing even without a subsequent fetch message (#4434).
• Fix for stored offsets not being committed if they lacked the leader epoch (#4442).
• Upgrade OpenSSL to v3.0.11 (while building from source) with various security fixes, check the release notes (#4454, started by @migarc1).
• Fix to ensure permanent errors during offset validation continue being retried and don't cause an offset reset (#4447).
• Fix to ensure max.poll.interval.ms is reset when rd_kafka_poll is called with consume_cb (#4431).
• Fix for idempotent producer fatal errors, triggered after a possibly persisted message state (#4438).
• Fix rd_kafka_query_watermark_offsets continuing beyond timeout expiry (#4460).
• Fix rd_kafka_query_watermark_offsets not refreshing the partition leader after a leader change and subsequent NOT_LEADER_OR_FOLLOWER error (#4225).

Upgrade considerations

• retry.backoff.ms: If it is set greater than retry.backoff.max.ms which has the default value of 1000 ms then it is assumes the value of retry.backoff.max.ms. To change this behaviour make sure that retry.backoff.ms is always less than retry.backoff.max.ms. If equal then the backoff will be linear instead of exponential.

• topic.metadata.refresh.fast.interval.ms: If it is set greater than retry.backoff.max.ms which has the default value of 1000 ms then it is assumes the value of retry.backoff.max.ms. To change this behaviour make sure that topic.metadata.refresh.fast.interval.ms is always less than retry.backoff.max.ms. If equal then the backoff will be linear instead of exponential.

Fixes

General fixes

• An assertion failed with insufficient buffer size when allocating rack information on 32bit architectures. Solved by aligning all allocations to the maximum allowed word size (#4449).
• The timeout for rd_kafka_query_watermark_offsets was not enforced after making the necessary ListOffsets requests, and thus, it never timed out in case of broker/network issues. Fixed by setting an absolute timeout (#4460).

Idempotent producer fixes

• After a possibly persisted error, such as a disconnection or a timeout, next expected sequence used to increase, leading to a fatal error if the message wasn't persisted and the second one in queue failed with an OUT_OF_ORDER_SEQUENCE_NUMBER. The error could contain the message "sequence desynchronization" with just one possibly persisted error or "rewound sequence number" in case of multiple errored messages. Solved by treating the possible persisted message as not persisted, and expecting a DUPLICATE_SEQUENCE_NUMBER error in case it was or NO_ERROR in case it wasn't, in both cases the message will be considered delivered (#4438).

Consumer fixes

• Stored offsets were excluded from the commit if the leader epoch was less than committed epoch, as it's possible if leader epoch is the default -1. This didn't happen in Python, Go and .NET bindings when stored position was taken from the message. Solved by checking only that the stored offset is greater than committed one, if either stored or committed leader epoch is -1 (#4442).
• If an OffsetForLeaderEpoch request was being retried, and the leader changed while the retry was in-flight, an infinite loop of requests was triggered, because we weren't updating the leader epoch correctly. Fixed by updating the leader epoch before sending the request (#4433).
• During offset validation a permanent error like host resolution failure would cause an offset reset. This isn't what's expected or what the Java implementation does. Solved by retrying even in case of permanent errors (#4447).
• If using rd_kafka_poll_set_consumer, along with a consume callback, and then calling rd_kafka_poll to service the callbacks, would not reset max.poll.interval.ms. This was because we were only checking rk_rep for consumer messages, while the method to service the queue internally also services the queue forwarded to from rk_rep, which is rkcg_q. Solved by moving the max.poll.interval.ms check into rd_kafka_q_serve (#4431).
• After a leader change a rd_kafka_query_watermark_offsets call would continue trying to call ListOffsets on the old leader, if the topic wasn't included in the subscription set, so it started querying the new leader only after topic.metadata.refresh.interval.ms (#4225).

Checksums

Release asset checksums:

• v2.3.0.zip SHA256 15e77455811b3e5d869d6f97ce765b634c7583da188792e2930a2098728e932b
• v2.3.0.tar.gz SHA256 2d49c35c77eeb3d42fa61c43757fcbb6a206daa560247154e60642bcdcc14d12