librdkafka v0.9.4 is a small feature release with a horde of enhancements and fixes.

NOTE: The librdkafka C++ API is unfortunately not ABI safe (the API stability is guaranteed though): C++ users will need to recompile their applications when upgrading librdkafka.

New features

• SASL: Windows SASL GSSAPI support using native Windows SSPI (#888)
• Added rd_kafka_offsets_for_times (KIP-79, #842)
• Added rd_kafka_producev() using va-args to construct message fields, including timestamps (#858, #707, #908, #345)
• C++: added produce(..., timestamp, ..)
• Added support for partition-specific queues (@lambdaknight)
  • allows redirecting partitions to separate queues and separates application threads.
• True low-latency (on linux and osx) by fd-based queue wakeups

Critical fixes

• leader failover error detection and recovery for both producer and consumer
• cgrp: failover robustness improvements
• cgrp: certain failed Heartbeats could mask out future heartbeat requests
• buffer handle callbacks could previously be triggered on wrong thread

Enhancements

• Events: expose EVENT_OFFSET_COMMIT
• Experimental cmake support (thanks to @ruslo)
• Validate topics in subscribe() (mainly regex validation)
• Added set_log_queue() - allows application thread log handling (#355)
• Add support for Metadata v1..2 (part of KIP-4)
• New metadata request and caching framework:
  • much fewer metadata requests
  • use cached metadata when deemed possible
  • KIP-4 support for lean metadata requests in clusters with many topics
  • improved (faster, more reliable) leader lookups
• C++: added TopicPartition::create(, ..offset)
• C++: added const attribute to TopicPartition getters
• C++: added TopicPartition::destroy(vector) helper
• C++: added commitSync(.., OffsetCommitCb) variant
• C++: Enrichment of Conf getter for callbacks. (#883, @vin-d)
• Expose rd_kafka_topic_partition_list_sort()
• Added max.in.flight -> max.in.flight.requests.per.connection alias
• Win32: add lz4 support (using NuGet lz4 package)
• Add cgrp stats: rebalance_age, rebalance_cnt, assignment_size

Fixes

• increase maximum supported topic limit to 1M
• fix static linking on osx
• Fix infinte wait in .._commit_queue() if callback on temp queue was used
• Improved handling of assign() and outstanding rebalance_cb on termination
• commit_queue: avoid global offset_commit_cb if local specified
• Imrpoved Heartbeat error handling
• Improve OffsetCommit failure handling
• Query topics with missing or down leaders from topics_scan()
• Handle rebalance from consumer_close (#1002)
• rd_kafka_producev() would send 0 timestamp by default instead of current time
• queue.buffering.max.ms and fetch.error.backoff.ms now takes precedence over socket.blocking.max.ms (#966)
• flush() fix: msg_cnt not initialized
• rdkafka_performance: fix multi -p partitions allocation error and memory leak
• Refactor query_watermark_offsets to use new leader lookup code
• Proper error propagation on invalid topic names (#977)
• callbacks will no longer cut consumer_poll() timeouts short
• improved queue handling and serving
• rd_kafka_producev(): fix topic/rkt referencing
• Fix ApiVersionRequest timeout (10s)
• rd_kafka_q_concat: fix erroneous unlock() on enq failure
• Speed up termination by emitting TERMINATE ops
• cgrp: enforce metadata update check if matched subscribed topics is empty
• Proper suppression of connect failure logs (#847)
• Proper ATOMIC_OPn (#708)
• fix: use out of scope variable (#974, @HeChuanXUPT)
• C++: Fix undefined references to static const ints (#961, #962, @bbfgelman1)
• mklove: lib_check now attempts compilation after pkg-config passed (for false positive of SSL on OSX)
• mklove: fix ./configure --reconfigure
• stats: fix brokers.toppars key duplication