Confluent's Python client for Apache Kafka v1.0.0

confluent-kafka-python is based on librdkafka v1.0.0, see the librdkafka v1.0.0 release notes for a complete list of changes, enhancements and fixes and upgrade considerations.

v1.0.0 is a major feature release:

• Idempotent producer - guaranteed ordering, exactly-once producing) support.
• Sparse/on-demand connections - connections are no longer maintained to all brokers in the cluster.
• KIP-62 - max.poll.interval.ms support in the Consumer.

This release also changes configuration defaults and deprecates a set of configuration properties, make sure to read the Upgrade considerations section below.

Upgrade considerations (IMPORTANT)

Configuration default changes

The following configuration properties have changed default values, which may require application changes:

• acks(alias request.required.acks) now defaults to all; wait for all in-sync replica brokers to ack. The previous default, 1 , only waited for an ack from the partition leader. This change places a greater emphasis on durability at a slight cost to latency. It is not recommended that you lower this value unless latency takes a higher precedence than data durability in your application.

• broker.version.fallback now to defaults to 0.10, previously 0.9. broker.version.fallback.ms now defaults to 0. Users on Apache Kafka <0.10 must set api.version.request=false and broker.version.fallback=.. to their broker version. For users >=0.10 there is no longer any need to specify any of these properties.

• enable.partition.eof now defaults to false. KafkaError._PARTITION_EOF was previously emitted by default to signify the consumer has reached the end of a partition. Applications which rely on this behavior must now explicitly set enable.partition.eof=true if this behavior is required. This change simplifies the more common case where consumer applications consume in an endless loop.

group.id is now required for Python consumers.

Deprecated configuration properties

The following configuration properties have been deprecated. Use of any deprecated configuration property will result in a warning when the client instance is created. The deprecated configuration properties will be removed in a future release.

librdkafka:

• offset.store.method=file is deprecated.
• offset.store.path is deprecated.
• offset.store.sync.interval.ms is deprecated.
• produce.offset.report is no longer used. Offsets are always reported.
• queuing.strategy was an experimental property that is now deprecated.
• reconnect.backoff.jitter.ms is no longer used, see reconnect.backoff.ms and reconnect.backoff.max.ms.
• socket.blocking.max.ms is no longer used.
• topic.metadata.refresh.fast.cnt is no longer used.

confluent_kafka:

• default.topic.config is deprecated.
• `CachedSchemaRegistryClient: url: was str, now conf dict with all application config properties

Idempotent Producer

This release adds support for Idempotent Producer, providing exactly-once producing and guaranteed ordering of messages.

Enabling idempotence is as simple as setting the enable.idempotence configuration property to true.

There are no required application changes, but it is recommended to add support for the newly introduced fatal errors that will be triggered when the idempotent producer encounters an unrecoverable error that would break the ordering or duplication guarantees.

See Idempotent Producer in the manual and the Exactly once semantics blog post for more information.

Sparse connections

In previous releases librdkafka would maintain open connections to all brokers in the cluster and the bootstrap servers.

With this release librdkafka now connects to a single bootstrap server to retrieve the full broker list, and then connects to the brokers it needs to communicate with: partition leaders, group coordinators, etc.

For large scale deployments this greatly reduces the number of connections between clients and brokers, and avoids the repeated idle connection closes for unused connections.

Sparse connections is on by default (recommended setting), the old behavior of connecting to all brokers in the cluster can be re-enabled by setting enable.sparse.connections=false.

See Sparse connections in the manual for more information.

Original issue librdkafka #825.

KIP-62 - max.poll.interval.ms is enforced

This release adds support for max.poll.interval.ms (KIP-62), which requires the application to call consumer.poll() at least every max.poll.interval.ms. Failure to do so will make the consumer automatically leave the group, causing a group rebalance, and not rejoin the group until the application has called ..poll() again, triggering yet another group rebalance. max.poll.interval.ms is set to 5 minutes by default.

Enhancements

• OpenSSL version bumped to 1.0.2r
• AvroProducer now supports encoding with fastavro (#492)
• Simplify CachedSchemaRegistryClient configuration with configuration dict for application configs
• Add Delete Schema support to CachedSchemaRegistryClient
• CachedSchemaRegistryClient now supports HTTP Basic Auth (#440)
• MessageSerializer now supports specifying reader schema (#470)

Fixes

• Fix crash when calling Consumer.consume without setting group.id(now required)
• CachedSchemaRegistryClient handles get_compatibility properly

Build/installation/tooling

• Integration tests moved to docker-compose to aid in cluster set-up/tear-down
• Runner script ./tests/run.sh added to simplify unit and integration test execution