librdkafka v1.0.0 release

v1.0.0 is a major feature release:

• Idempotent producer - guaranteed ordering, exactly-once producing.
• Sparse/on-demand connections - connections are no longer maintained to all brokers in the cluster.
• KIP-62 - max.poll.interval.ms for high-level consumers

This release also changes configuration defaults and deprecates a set of configuration properties, make sure to read the Upgrade considerations section below.

Upgrade considerations (IMPORTANT)

librdkafka v1.0.0 is API (C & C++) and ABI (C) compatible with older versions of librdkafka, but there are changes to configuration properties that may require changes to existing applications.

Configuration default changes

The following configuration properties have changed default values, which may require application changes:

• acks (alias request.required.acks) default is now all (wait for ack from all in-sync replica brokers), the previous default was 1 (only wait for ack from partition leader) which could cause data loss if the leader broker goes down.
• enable.partition.eof is now false by default. Applications that rely on ERR__PARTITION_EOF to be emitted must now explicitly set this property to true. This change was made to simplify the common case consumer application consume loop.
• broker.version.fallback was changed from 0.9 to 0.10 and broker.version.fallback.ms was changed to 0. Users on Apache Kafka <0.10 must set api.version.request=false and broker.version.fallback=.. to their broker version. For users >=0.10 there is no longer any need to specify any of these properties. See https://github.com/edenhill/librdkafka/wiki/Broker-version-compatibility for more information.

Deprecated configuration properties

• topic.metadata.refresh.fast.cnt is no longer used.
• socket.blocking.max.ms is no longer used.
• reconnect.backoff.jitter.ms is no longer used, see reconnect.backoff.ms and reconnect.backoff.max.ms.
• offset.store.method=file is deprecated.
• offset.store.path is deprecated.
• offset.store.sync.interval.ms is deprecated.
• queuing.strategy was an experimental property that is now deprecated.
• msg_order_cmp was an experimental property that is now deprecated.
• produce.offset.report is no longer used. Offsets are always reported.
• auto.commit.enable (topic level) for the simple (legacy) consumer is now deprecated.

Use of any deprecated configuration property will result in a warning when the client instance is created. The deprecated configuration properties will be removed in a future version of librdkafka. See issue #2020 for more information.

Configuration checking

The checks for incompatible configuration has been improved, the client instantiation (rd_kafka_new()) will now fail if incompatible configuration is detected.

max.poll.interval.ms is enforced

This release adds support for max.poll.interval.ms (KIP-62), which requires the application to call rd_kafka_consumer_poll()/rd_kafka_poll() at least every max.poll.interval.ms. Failure to do so will make the consumer automatically leave the group, causing a group rebalance, and not rejoin the group until the application has called ..poll() again, triggering yet another group rebalance. max.poll.interval.ms is set to 5 minutes by default.

Idempotent Producer

This release adds support for Idempotent Producer, providing exactly-once producing and guaranteed ordering of messages.

Enabling idempotence is as simple as setting the enable.idempotence configuration property to true.

There are no required application changes, but it is recommended to add support for the newly introduced fatal errors that will be triggered when the idempotent producer encounters an unrecoverable error that would break the ordering or duplication guarantees.

See Idempotent Producer in the manual and the Exactly once semantics blog post for more information.

Sparse connections

In previous releases librdkafka would maintain open connections to all brokers in the cluster and the bootstrap servers.

With this release librdkafka now connects to a single bootstrap server to retrieve the full broker list, and then connects to the brokers it needs to communicate with: partition leaders, group coordinators, etc.

For large scale deployments this greatly reduces the number of connections between clients and brokers, and avoids the repeated idle connection closes for unused connections.

See Sparse connections in the manual for more information.

Original issue #825.

Features

• Add support for ZSTD compression (KIP-110, @mvavrusa. Caveat: will not currently work with topics configured with compression.type=zstd, instead use compression.type=producer, see #2183)
• Added max.poll.interval.ms (KIP-62, #1039) to allow long processing times.
• Message Header support for C++ API (@davidtrihy)

Enhancements

• Added rd_kafka_purge() API to purge messages from producer queues (#990)
• Added fatal errors (see ERR__FATAL and rd_kafka_fatal_error()) to raise unrecoverable errors to the application. Currently only triggered by the Idempotent Producer.
• Added rd_kafka_message_status() producer API that may be used from the delivery report callback to know if the message was persisted to brokers or not. This is useful for applications that want to perform manual retries of messages, to know if a retry could lead to duplication.
• Backoff reconnects exponentially (See reconnect.backoff.ms and reconnect.backoff.max.ms).
• Add broker[..].req["reqType"] per-request-type metrics to statistics.
• CONFIGURATION.md: Added Importance column.
• ./configure --install-deps (and also --source-deps-only) will automatically install dependencies through the native package manager and/or from source.

Fixes

General

• rd_kafka_version() was not thread safe
• Round up microsecond->millisecond timeouts to 1ms in internal scheduler to avoid CPU-intensive busy-loop.
• Send connection handshake requests before lower prio requests.
• Fix timespec conversion to avoid infinite loop (#2108, @boatfish)
• Fix busy-loop: Don't set POLLOUT (due to requests queued) in CONNECT state (#2118)
• Broker hostname max size increased from 127 to 255 bytes (#2171, @Vijendra07Kulhade)

Consumer

• C++: Fix crash when Consumer ctor fails
• Make sure LeaveGroup is sent on unsubscribe and consumer close (#2010, #2040)
• Remember assign()/seek():ed offset when pause()ing (#2105)
• Fix handling of mixed MsgVersions in same FetchResponse (#2090)

Producer

• Added delivery.timeout.ms -> message.timeout.ms alias
• Prevent int overflow while computing abs_timeout for producer request… (#2050, @KseniyaYakil).
• Producer: fix re-ordering corner-case on retry.

Windows

• win32: cnd_timedwait*() could leave the cond signalled, resulting in high CPU usage.

Build/installation/tooling

• Makefile: fix install rule (#2049, @pacovn)
• Fixing Counting error in rdkafka_performance #1542 (#2028, @gnanasekarl)
• OpenSSL 1.1.0 compatibility (#2000, @nouzun, @wiml)
• Set OpenSSL locking callbacks as required, dont call CRYPTO_cleanup_all_ex_data (#1984, @ameihm0912)
• Fix 64-bit IBM i build error (#2017, @ThePrez)
• CMake: Generate pkg-config files (@Oxymoron79, #2075)
• mklove: suggest brew packages to install on osx
• rdkafka_performance: Add an option to dump the configuration (@akohn)
• Check for libcrypto explicitly (OSX Mojave, #2089)