Confluent's Python client for Apache Kafka

confluent-kafka-python is based on librdkafka v1.2.0, see the librdkafka v1.2.0 release notes for a complete list of changes, enhancements, fixes and upgrade considerations.

• Transaction aware consumer (isolation.level=read_committed) implemented by @mhowlett.
• Sub-millisecond buffering (linger.ms) on the producer.
• Improved authentication errors (KIP-152)

Consumer-side transaction support

This release adds consumer-side support for transactions. In previous releases, the consumer always delivered all messages to the application, even those in aborted or not yet committed transactions. In this release, the consumer will by default skip messages in aborted transactions. This is controlled through the new isolation.level configuration property which defaults to read_committed (only read committed messages, filter out aborted and not-yet committed transactions), to consume all messages, including for aborted transactions, you may set this property to read_uncommitted to get the behaviour of previous releases. For consumers in read_committed mode, the end of a partition is now defined to be the offset of the last message of a successfully committed transaction (referred to as the 'Last Stable Offset'). For non-transactional messages there is no change from previous releases, they will always be read, but a consumer will not advance into a not yet committed transaction on the partition.

Upgrade considerations

• linger.ms default was changed from 0 to 0.5 ms to promote some level of batching even with default settings.

New configuration properties

• Consumer property isolation.level=read_committed ensures the consumer will only read messages from successfully committed producer transactions. Default is read_committed. To get the previous behaviour, set the property to read_uncommitted, which will read all messages produced to a topic, regardless if the message was part of an aborted or not yet committed transaction.

Enhancements

• Cache FastAvro schema for improved Avro Serialization/Deserialization (@BBM89, #627)
• Protocol decoding optimization, increasing consume performance.
• Add CachedSchemaRegistry docs (@lowercase24 , #495)

Fixes

General:

• Rate limit IO-based queue wakeups to linger.ms, this reduces CPU load and lock contention for high throughput producer applications. (#2509)
• SSL: Use only hostname (not port) when valid broker hostname (by Hunter Jacksson)
• SSL: Ignore OpenSSL cert verification results if enable.ssl.certificate.verification=false (@salisbury-espinosa)
• SASL Kerberos/GSSAPI: don't treat kinit ECHILD errors as errors (@hannip)
• Refresh broker list metadata even if no topics to refresh (#2476)
• Correct AdminClient doc (@lowercase24, #653)
• Update Avro example to be compliant with csh (@andreyferriyan , #668)
• Correct Avro example typo (@AkhilGNair, #598)

Consumer:

• Make pause|resume() synchronous, ensuring that a subsequent poll() will not return messages for the paused partitions.
• Consumer doc fixes (@hrchu , #646, #648)

Producer:

• Fix message timeout handling for leader-less partitions.
• message.timeout.ms=0 is now accepted even if linger.ms > 0 (by Jeff Snyder)