Maintenance release

@skarlsson

Maintenance release

Enhancements

Increase .._TOPICS_MAX to 1M
Add cgrp stats: rebalance_age, rebalance_cnt, assignment_size
win32: allow the use of LIBRDKAFKA_STATICLIB to enable static library (#905, @skarlsson)
C++: Add support for setting "consume_cb" in Conf with a RdKafka::ConsumeCb and proper handling of that callback object by the underlying C library. (#741, @lambdaknight)
Added socket.nagle.disable to disable the Nagle algorithm (#862)
Added rd_kafka_topic_partition_list_sum()
Added connect_cb and closesocket_cb to conf

Fixes

Fix relative offset calculation (#859, @yunjing)
cgrp: dont get stuck in wait-coord state on reoccuring query errors (issue #844)
cgrp: fix a number of assign() problems (issues #871, #844, possibly #730)
cgrp: increased robustness for wait_commit et.al (#785)
cgrp: time out ops in wait_coord queue (#785)
cgrp: do async termination by state to avoid recursive locks from timeout_scan
Dont store initial offset in fetch_start0 (#742)
Win32 compilation and linking fixes
queue.buffering.max.kbytes wrapped on platforms with 32-bit size_t (#998)
Timeout was not enforced in flush()
librdkafka++: just link C library, not also its dependencies
rdkafka_performance: fix multi -p partitions allocation error and memory leak
Fix ApiVersionRequest timeout (10s)
rd_kafka_q_concat: fix erroneous unlock() on enq failure
C++: Added TopicPartition::create(, ..offset)
C++: Fix undefined references to static const ints (#961, #962, @bbfgelman1)
Dont start SASL for the internal broker (#968)
Proper suppression of connect failure logs (#847)
Proper ATOMIC_OPn (#708)
fix: use out of scope variable (#974, @HeChuanXUPT)
INTRODUCTION: fix wrong batch.num.messages default value (#924)
configure: check for library headers too (fixes OpenSSL failures on OSX)
mklove: lib_check now attempts compilation after pkg-config passed (for false positive of SSL on OSX)
mklove: fix ./configure --reconfigure
stats: fix brokers.toppars key duplication
consumer: Correctly set timestamp for compressed messages (#858)
Dont attempt SASL auth with broker-unsupported mechs (issue #907)
consumer_poll() without a valid group.id would return a broken message_t
Fix duplicate desired_link calls (crash or memory leak)
Fix for async offset commit issue on AIX (#852, @asharma339)
_builtin.. replacements for native toolchains on sun and aix (#865, @asharma339)
Add snappy function prefix to avoid symbol collision with google snappy (#880, @hustlijian)
toppars: Dont mix up UNKNOWN / DESIRED flags
Use compiler specific atomic ops for sun studio compiler (#869, @asharma339)
fix bug rd_kafka_timer_t microsecond interval overflow. (#870, @lvxv)
Set HeartbeatRequest timeout to session.timeout.ms
Allow max one HeartbeatRequest in transit
DescribeGroups: Avoid zero-byte allocations (for AIX)
fix func rd_kafka_new declare use_conf but not use cause conf->socket_cb segmentfault (#863) (@wadee)

@takano-akio

Additions

Event interface (see rd_kafka_event_type_t)
VerifiableConsumer and VerifiableProducer clients for running official kafkatests
Added support for LZ4 compression (KIP-57)
Added rd_kafka_commit_queue()
Added rd_kafka_queue_length()
Added rd_kafka_flush()
Added IO FD-based queue triggering for including librdkafka in IO-based poll loops (issue #749)
Added enable.partition.eof to allow turning off PARTITION_EOF
auto.offset.reset: added aliases: beginning and end
Added experimental blocking produce call (.._F_BLOCK flag to produce())

Enhancements

rd_kafka_version_str(): return a git-aware version string
Retry metadata requests if returned response is empty
Expedite MetadataRequests in front of other requests
Improved ...metadata() selection of broker to query
Added ERR__TIMED_OUT_QUEUE to differentiate between in-transit and in-queue timeouts
rdkafka_performance: added -N to disable delivery reports
rdkafka_performance: added -G for high-level KafkaConsumer
Regex support on Win32: added fallback regex implementation (issue #552)
Producer: Allow unlimited number of messages per batch/MessageSet (bump batch.num.messages to 10000)
Producer: Copy small message payloads to transmit buffer (issue #788)
.._conf_set(): revert to default value (if any) when passed value==NULL.
Redesigned toppar op version barriers to be more robust
Automatically enable api.version.request for broker.version.fallback>=0.10
Expedite termination by broadcasting state change on destroy
Improve ALL_BROKERS_DOWN event generation
Fall back to mutex-protected variables when 32 or 64 bit atomics are not available (#693)
Disable SSLv3

Consumer fixes

Fix crash in committed() with short timeouts (issue #827)
Dont attempt offset commits for empty non-null list
cgrp: clear stored member_id on UNKNOWN_MEMBER_ID errors
handle_JoinGroup: dont propagate UnknownMemberId to app
cgrp term: force final unassign call even if waiting for rebalance
Dont send Heartbeats when calling rebalance_cb(REVOKE)
cgrp: ignore outdated Join & Sync responses when state has transitioned
Dont propagate error for empty commits when internal commits are retried
Stop offset_query_tmr when stopping fetcher
OffsetCommit&OffsetFetch request: properly coalesce partitions for the same topic (issue #785)
C++: added Consumer.seek() (#699)
Fix lock ordering issue in fetcher
Update subscription on topic changes (issues #802, #644, #576, #461)
Fixed a number of consumer (group) corner-cases
Resume GroupCoord queries on all errors (#768)
Error out on unknown partition.assignment.strategy:s
Maintain cgrp version barrier for relevant internal request ops (fixes #761)
Dynamically increase the max fetch size when needed (#597, by @takano-akio)
Proper timeout calculation for consume(), consumer_poll(), etc
Properly handle empty assignments in SyncGroupResponse (#603)
Fixed a number of hang-on-termination issues
Consumers could stall in offset-wait state on leader change for up to socket.timeout.ms * retries (#843)

Producer fixes

Send uncompressed if compression messageset is larger than uncompressed (issue #781)
Proper ProducerResponse error checking before retries (issue #777)
Time out messages on toppar xmit msgq too (fixes #776, #672)
Dont create cgrp for producers if group.id is configured (issue #691)
Fixed a number of hang-on-termination issues

General fixes

Bring down the connection regardless of socket.max.fails if in-flight requests time out (#756)
Count partially sent buffers as in-flight when timeout scanning (issue #756)
Dont insert requests in front of partly sent SSL requests
Reset partially sent buffer's offsets to 0 on disconnect (issue #756, #792)
Dont prepend FLASH messages before partially sent buffers (possibly #756)
Time out broker retry bufs (issue #776)
Count topic_t app refcounts separately (issues #345, #821)
Clean up OpenSSL global locks when all librdkafka instances are gone (issue #815)
Fixed SSL broker_fail() arguments (issue #796)
Improved purging of partition op & message queues to speed up termination
Proper thread creation error propagation (issue #712)
Forwarded queue's content was not properly moved to dest queue on fwd_set()
Avoid endless loop when reconnect.backoff.jitter.ms is low (#779)
Fix double free of topic.blacklist (issue #778)
Fix SASL PLAIN auth (broken in 0.9.1)
rdkafka_performance: could exit before all msgs sent when host was slow
query_watermark_offsets() workaround for KAFKA-1588 (issue #700)
Proper shrinking of rktparlist (issue #739 reported by @diazdiego86)
query_watermark: Trigger leader query if leader not known (closes #694, fix by @JohanAR)
Broadcast broker state changes (et.al) to expedite waiter wakeups (closes #695)
Properly handle empty sasl.kerberos.kinit.cmd (seen in #688)
rd_kafka_new: delete conf only on success (#836 by @julien-lecomte)

@eugpermar

New major features

Broker-based balanced High-Level KafkaConsumer (broker version >= 0.9)
SASL support (client authentication) - requires Cyrus libsasl
SSL support (client authentication, transport encryption) - requires OpenSSL
Native Win32 support (Visual Studio)
Broker-based quota support (KIP-13)
Relative offsets support (KIP-31)
Message timestamp support (KIP-32)

New enhancements

Broker API version query support for brokers >=0.10.0 with fallbacks for older brokers. (KIP-35)
- api.version.request - enable/disable ApiVersionRequests
- api.version.fallback.ms - how long to use the fallback version after ApiVersionRequest failure
- broker.version.fallback - fallback version to adapt API usage to.
- More info: https://github.com/edenhill/librdkafka/wiki/Broker-version-compatibility
Added broker connection max in-flight enforcement: max.in.flight.requests.per.connection
Added config aliases "retries", "acks" (#157)
request.required.acks/acks: added support for "all" (-1)
Flag config properties now support explicit +addx,-dely and "all"
Added rd_kafka_last_error() alternative to errno for legacy APIs (#571)
Added rd_kafka_get_err_descs() to retrieve full list of error codes.
Added rd_kafka_partitions_pause/_resume()
- Consumer: flush local fetchq for partition(s) and stop fetching
- Producer: dont send any ProduceRequests to broker for partition(s)
  This is also used when a rebalance_cb is registered between the time of
  rebalancing and until the application has called assign() to synchronized state.
  Prevents app from receiving messages after a rebalance but before assign()
Keep track of app_offset: last message+1 provided to application (issue #106). This is the "next offset" for the application.
Added log.connection.close to turn off logging of broker-triggered disconnects (issue #437)
Added rd_kafka_mem_free()
Formal C and C++ API docs (doxygen)
Added RdKafka::Message::topic_name().
Allows retreiving message topic (name) for auto consumed topics where we dont have a local Topic object
Add C++ Metadata API (@eugpermar and @incessantier)
Changed topic.metadata.refresh.interval.ms to 5m, must be lower than server idle timeout
Alias some new java client properties (bootstrap.servers, enable.auto.commit)
Added topic-level "compression.codec" configuration
Added "builtin.features" property to allow applications to query for built-in features (e.g., gzip, snappy, zlib)
Add Consumer::OffsetTail(). (@bwelling)
Removed enforce.isr.cnt - should be handled by broker now and request.required.acks=-1
Added rd_kafka_topic_opaque()
Added rd_kafka_*_conf_get() for querying conf property values on a conf object
New high-level balanced KafkaConsumer (requires broker version >= 0.9)
Add consumer queue support for C++. (@rthalley and @macisamuele)
Fixed alignment errors on Sparc, et.al.
Increased topic.metadata.refresh.interval.ms to the official default 10 minutes (#363)
Added config property 'socket.blocking.max.ms' to control max socket poll time
Added lazy producer message offset reporting
When 'produce.offset.report' is off, produced offsets will still be reported or the last message in each produced batch.
This means an application can still keep track of offsets (though a bit more roughly) without the cost of per-message offset reports.
Added rd_kafka_yield() to break out of callback loops
Propagate UNKNOWN_TOPIC and UNKNOWN_PARTITION to consumers (issue #348)
Handle broker name and nodeid updates (issue #343)
This also migrates bootstrap brokers to proper brokers if they can be exactly matched by name.
Added consume.callback.max.messages (issue #248)
Added C++ Consumer::consume_callback as front-end of rd_kafka_consume_callback (kodekarl)
Add a built in consistent hashing partitioner (@NZKoz)
Add ability to specify a CRL for checking remote certificates (@vincentbernat)

Fixes

Preserve configured partition.assignment.strategy order
Final offset commits on consumer close() are now limited to session.timeout.ms
rd_kafka_log_print should print to stderr, not stdout
C++ unassign() assigned an empty list, which is not an unassign..
Dont fail/retry unsent requests on broker failure, leave them in queue.
Portability fixes (FreeBSD, Win32, MUSL libc, SunOS, AIX)
C++: added virtual destructors to all classes that have a virtual function (@danny794)
Consumer: fix race condition with start(logical_offset) quickly followed by seek(abs_offset)
Maintain per-partition op-version in Fetch requests (issue #586)
Richer CONFIGURATION.md format (supported values, value type)
Fixed a number of possible hang-on-termination scenarios.
Improved handling of failed broker requests (exposed by #529)
Fixed cases where a timeout_ms of 0 would serve precisely timed events forever
Fixed buffer retry counting
Dont rely on partition leader broker to be set when querying metadata (issue #518)
Ignore intermittent Topic Metadata failures while brokers are rebalancing. (issue #513)
rd_kafka_topic_partition_available() returned 1 for leader-less partitions (issue #513). Regression since 0.8.6
Backoff sleep on broker UP -> DOWN state transition was never called. This could trigger busy-looping when connecting to a broken broker.
Added configurable reconnect delay with jitter: reconnect.backoff.jitter.ms (issue #403)
Snappy endian swapping for older glibc 2.9 (issue #505, #407)
Fix possible crash on buf retries when broker is unavailable
Consumer errors are now rkmessage-based rather than error_cb
Added RD_KAFKA_OFFSET_INVALID to indicate invalid offsets in various APIs
Minimize default address caching time to 1s (issue #494)
Added rd_kafka_errno() (issue #489)
Proper generation and installing of pkg-config .pc files for both C and C++ libraries
Report unsupported compression codecs (on consume) as CONSUMER_ERR ops
RdKafka::Conf->set "socket_cb" and "open_cb" checked for wrong Conf type
Deprecate rd_kafka_set_logger() in favour of rd_kafka_conf_set_log_cb()
Dont reset partition count to zero on metadata topic error
A metadata partition count of 0 is returned by the broker with errors such as "No leader broker" for a topic, since these errors are
typically intermittent we hold on to the existing partition count.
Emit ALL_BROKERS_DOWN if none of the initial bootstrap brokers could be added.
Consumer offset fixes (#352, #276, #227)
Dont loose messages when partition goes away (and there is no UA partition)
Previously any messages in a partition's queue would be silently lost if a partition went away (and there was no UA partition).
These messages are now properly delivery reported with UNKNOWN_PARTITION.
Provide rkmessage->rkt topic in dr_msg_cb (#333)
Fixed wait in rd_kafka_consume_callback_queue().
It returned immediately if no messages were available (timeout_ms was not used).
Fix occasional invalid json on statistics buffers (@alfred-landrum)
dr_msg_cb: rkmessage->partition is now properly set (issue #233)
Renamed position() -> committed() and added position() that returns next offset. According to the Java client.
Workaround Java client and broker Offset metadata parsing bugs (issue #577)
Added rd_kafka_query_watermark_offsets() and rd_kafka_get_watermark_offsets() (issue #503, #530)
C++: Added commitSync and commitAsync APIs taking a list of offsets (issue #528)
Added enable.auto.offset.store (issue #528)
sasl.kerberos.kinit.cmd now takes a command+args string template (issue #635)
Send consumer group user_data as empty instead of NULL to workarund broker 0.9.0.0 bug (issue #660)

@secretmike

Mainly a maintenance release with 15 enhancements and 25 fixes.

Enhancements

NULL messages (and keys) are now supported (@secretmike)
Switched to use zlib crc functions for increased performance (@tburmeister)
Added 'group.id' config property to topic config (overrides global 'group.id')
Include group.id in offset file (issue #199). Escape offset filenames.
C++: Implemented Message::key() (@KodeKarl).
Enabled OSX builds for travis-ci
Added "consumer_lag", "lo_offset", "hi_offset", "stored_offset" to stats output.
Added conf: delivery.report.only.error: only trigger dr_cb() for errored messages.
Added "internal.termination.signal" property (default off)
rdkafka_performance: added latency measurement mode (-l) and offset reporting (-O)
rdkafka_performance: Added '-A ': write per-message latency to file
rdkafka_performance: send NULL key when -s 0
rdkafka_performance: print key
rdkafka_performance: Make -c work in consumer mode to limit number of messages consumed
rdkafka_performance: Print received message content when verbosity is >= 3

Fixes

Offset storage fixes (issue #191)
Extern C-ilize rdkafka.h (issue #216)
Revert request.required.acks limit (-1..1) back to (-1..1000) (issue #208)
Avoid excessive topic leader queries when a broker goes down
Renamed RdKafka::Producer::MSG_COPY and MSG_FREE to RK_MSG_COPY and RK_MSG_FREE (issue #198)
rdkafka_performance: Better usage error reporting
Avoid re-inclusion of rdkafkacpp.h (issue #195)
rdkafka_performance: proper stats_fp closing
Fix mismatches in broker_keep and broker_destroy that cause brokers to leak (#193) (@thelamb)
Silence recurring broker resolve failures (#186)
Added target dependencies for "make -j". (@KodeKarl)
Dont log broker signal interrupts when terminating
Fixed double-free of conf (defect added in ce8cd60)
mklove update for mktemp (issue #176)
Dont leak topic_conf memory when already added topic is passed to rd_kafka_topic_new()
NULL conf pointer after free (@oldmantaiter)
remove "executable" attribute from source files (@rgerhards)
Dont require auto.commit.enable for stored offsetting (issue #171)
Drop earlier (unwanted) messages from fetched messagesets
Increased message.max.retries to allow long-time offline queueing.
Library ordering (fixes Ubuntu ppc builds)
Require linkage
Make sure to store EOF offset on termination (issue #164)
If RD_KAFKA_OFFSET_TAIL() rewinds past offset 0, cap it to 0.
OSX requires linking required libs when creating dylib (issue #161)
Use proper .dylib install_name on OSX

Thanks to @tburmeister, @thelamb, @KodeKarl, @rgerhards, @oldmantaiter, @secretmike

Enhancements and bugfixes:

Dont link C++ library with C library - avoids symbol leakage
Fixed endian.h check (for kFreeBSD, thanks to paravoid)
rd_kafka_topic_new() was not completely thread-safe
Unlock timer lock when calling timer callback (and honour rk_terminate)
Store EOF offset for empty partitions on termination (previously did not store any offset)
Off-by-one error when calculating tv_nsec - by dalegaard
Fix metadata cache purge when topic.metadata.refresh.interval.ms=-1 (issue #156)
Allow 0 value for fetch.error.backoff.ms
Use a random broker for metadata requests
Dont crazy-spin on topic.metadata.refresh.interval.ms==-1 (issue #156)
Fixed _BIG_ENDIAN check on Solaris Sparc - thanks to shivashis
Now compiles on FreeBSD - thanks to rtyler
Sparse metadata retrieval mode (consumes less bandwidth) (issue #138)
Config property: topic.metadata.refresh.sparse = true or false
Added "protocol" debugging (sent and received message types)

The 0.8.4 release introduces the new C++ interface, a configure script for increased portability and a large number of enhancements and bug fixes.

New features and enhancement

Native C++ interface - see src-cpp/rdkafkacpp.h
./configure support using the mklove framework: https://github.com/edenhill/mklove
pkg-config rdkafka.pc file is now generated during build
Broker-based offset storage using OffsetCommit and OffsetFetch
Added Metadata API (issues #90 and #94)
rdkafka_example: Added -L (list) mode to list metadata
Added batch producer rd_kafka_produce_batch() for even more performant producing (issue #112)
Added queued.max.messages.kbytes (local consumer queue limit) (issue #81)
Added enforce.isr.cnt to locally enforce that this value is <= cluster ISR cnt (issue #91)
rdkafka_performance: Added -G <pipe command> to pipe stats output directly to command.
rdkafka_performance: Added -r <msgs/s> producer rate limiting
Added broker.address.family config property (limits broker connectivity to v4 or v6)
Added experimental msg_dr_cb() passing a complex message struct (NOT API SAFE just yet)
Added rd_kafka_opaque() to return the opaque pointer set by set_opaque
Added socket_cb and open_cb config properties (issue #77) (for CLOEXEC)
Added RPM package spec file for building RPM package - see rpm/librdkafka.spec
Added "msgs" to toppar stats output
rdkafka_performance: added tabular output, and more..
Solaris/SunOS support
Bart Warmerdam added an example how to integrate with ZooKeeper in examples/rdkafka_zookeeper_example.c
Support for infinite message timeouts message.timeout.ms=0 (thanks to Bart Warmerdam)
Added socket.keepalive.enable config property to enable SO_KEEPALIVE
Added socket.max.fails to disconnect from broker on request timeouts (defaults to 3)
Added produce.offset.report config property to propagate message offset of produced message back to application.
Added RD_KAFKA_OFFSET_TAIL(cnt) - start consuming cnt messages from end.
New consumer queue re-routing to pull messages from multiple topics+partitions simultaneously on the same thread with one single call. (rd_kafka_consume_start_queue())

Bug fixes

All signals are now properly blocked by rdkafka threads - no need to set up SIGPIPE handlers
in the application anymore.
Fixed compilation errors and warnings on various platforms
rdkafka_performance: use message.timeout.ms default value (issue #74)
Added redundant but correctly spelled "committed_offset" field to stats (issue #80)
Dont retry failed Fetch requests, wait for next intervalled Fetch (issue #84)
rd_kafka_*conf_dump() did not properly return all types of values (namely string2int mappings)
More consistent handling of unknown topics and partitions (issue #96)
Handle transient partition_cnt=0 during topic auto creation (issue #99)
Add pthread_attr_destroy() (thanks to Bart Warmerdam)
Consuming from a cluster with more than 500 partitions would crash rdkafka, this is now fixed. (issue #111)
Fixed MacOSX 10.9 compilation
Use .dylib extension for shared library on MacOSX
Fixed dead-lock race condition on metadata update (issue #131)
Topics were not destroyed in example code (issue #130)

The OpenSuse build service is now used to ensure portability, please see
https://build.opensuse.org/package/show/home:edenhill/librdkafka
for build results and packages.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Maintenance release

Enhancements

Fixes

Additions

Enhancements

Consumer fixes

Producer fixes

General fixes

New major features

New enhancements

Fixes

Enhancements

Fixes

New features and enhancement

Bug fixes

Sponsors

Releases: confluentinc/librdkafka

v0.9.3

Maintenance release

Enhancements

Fixes

0.9.2

Additions

Enhancements

Consumer fixes

Producer fixes

General fixes

0.9.1

New major features

New enhancements

Fixes

0.8.6

Enhancements

Fixes

0.8.5

0.8.4

New features and enhancement

Bug fixes

Sponsors

0.8.3

0.8.2

0.8.1

0.8.0