Feature release

v0.11.0 is a new feature release of librdkafka with support for the new Kafka message format (MsgVersion 2) which makes librdkafka (and any librdkafka-based clients) transparently compatible for use with the EOS (Exactly-Once-Semantics) supporting Java client released with Apache Kafka v0.11.0.

This release also includes enhancements and fixes as listed below.

NOTE: While librdkafka implements the new Message version and features, it does not yet implement the EOS (Exactly-Once-Semantics) functionality itself.

NOTE: The librdkafka C++ API is unfortunately not ABI safe (the API stability is guaranteed though): C++ users will need to recompile their applications when upgrading librdkafka.

Upgrade notes

api.version.request: The api.version.request property (see https://github.com/edenhill/librdkafka/wiki/Broker-version-compatibility) default value has changed from false to true, meaning that librdkafka will make use of the latest protocol features of the broker without the need to set the property to true explicitly on the client.

WARNING: Due to a bug in Apache Kafka 0.9.0.x, the ApiVersionRequest (as sent by the client when connecting to the broker) will be silently ignored by the broker causing the request to time out after 10 seconds. This causes client-broker connections to stall for 10 seconds during connection-setup before librdkafka falls back on the broker.version.fallback protocol features. The workaround is to explicitly configure api.version.request to false on clients communicating with &lt=0.9.0.x brokers.

Producer: The default value of queue.buffering.max.ms was changed from 1000ms to 0ms (no delay). This property denotes the internal buffering time (and latency) for messages produced.

Features

• Added support for MsgVersion v2 (message format of KIP-98) - message format compatible with EOS clients
• Added support for client interceptors
• Added support for dynamically loaded plugins (plugin.library.paths, for use with interceptors)
• Added SASL SCRAM support (KIP-84)
• Added builtin SASL PLAIN provider (for Win32, #982)

Enhancements

• Deprecate errno usage, use rd_kafka_last_error() instead.
• Deprecate rd_kafka_wait_destroyed().
• Implemented per-partition Fetch backoffs, previously all partitions for the given broker were backed off.
• Added updated Kafka protocol and error enums
• Added rd_kafka_message_latency()
• Added rd_kafka_clusterid() and rd_kafka_type()
• SSL: set default CA verify locations if ssl.ca.location is not specified
• C++: add yield() method
• Added support for stats as events (#1171)
• Build with system liblz4 if available, else fall back on built-in lz4, for improved portability.
• Use SNI when connecting through SSL (@vincentbernat)
• Improve broker thread responsiveness, decreasing internal latency
• Improve OpenSSL config error propagation (#1119)
• Prioritize all relevant user-facing ops (callbacks) over messages on poll queue (#1088)
• Added global->topic config fallthru: default topic config properties can now be set effortlessly on global config object.
• Log offset commit failures when there is no offset_commit_cb (closes #1043)
• Add CRC checking support to consumer (#1056)
• C++: Added seek() support to KafkaConsumer
• Added rd_kafka_conf_dup_filter() to selectively copy a config object.

Fixes:

• Avoid _ALIGN re-definition on BSD (#1225)
• rdkafka_performance: exit with code 1 if not all messages were delivered
• Fix endianism issues that were causing snappy to compress incorrectly (#1219, @rthalley)
• Fix stability on AIX (#1211)
• Document that rd_kafka_message_errstr() must not be used on producer
• Add support for re-queuing half-processed ops to honour yield()
• Handle null Protocol in JoinGroupResponse (#1193)
• Consumer: Proper relative offset handling (#1192, @rthalley)
• OSX: silence libsasl deprecated warnings
• partition count should be per topic in offset request buffer (closes #1199, @matthew-d-jones)
• fix build on SmartOS (#1186 by @misterdjules)
• ERR_remove_thread_state OpenSSL version checking
• Don't emit TIMED_OUT_QUEUE for timed out messages (revert)
• producev() default partition should UA, not 0 (#1153)
• Fix SaslHandshakeRequest timeout to 10s
• SASL: fix memory leak: received SASL auth frames were not freed
• Clean up OpenSSL per-thread memory on broker thread exit
• Properly auto-set metadata.max.age.ms when metadata.refresh.interval.ms is disabled (closes #1149)
• Fix memory alignment issues (#1150)
• configure: auto add brew openssl pkg-config path
• Fix consumer_lag calculation (don't use cached hi_offset)
• rdkafka_example: fix message_errstr usage, not allowed on producer
• Avoid use of partially destroyed topic object (#1125)
• Improve reconnect delay handling (#1089)
• C++: fix conf->get() allocation (closes #1118)
• Use app_offset to calculate consumer_lag (closes #1112)
• Fix retrybuf memory leak on termination when broker is down
• Fix small memory leak in metadata_leader_query
• Fix use-after-free when log.queue and debug was used
• consumer_example: fix crash on -X dump (closes #841)
• Added rd_kafka_offsets_store() (KafkaConsumer::offsets_store) (closes #826)
• Optimize broker id lookups (closes #523)
• Don't log broker failures when an error_cb is registered (closes #1055)
• Properly log SSL connection close (closes #1081)
• Win32 SASL GSSAPI: protection level and message size were not sent
• C++: improved error reporting from Conf::set()
• Flush partition fetch buffer on seek (from decide())
• Fix recursive locking on periodic refresh of leader-less partition (closes #1311)