Skip to content
zioPublic

About

A fast Kafka client for ZIO and ZIO Streams

Resources

Security policy

Stars

365 stars

Watchers

15 watching

Forks

Repository files navigation

ZIO Kafka

ZIO Kafka is a Kafka client for ZIO. It provides a purely functional, streams-based interface to the Kafka client and integrates effortlessly with ZIO and zio-streams. Often zio-kafka programs have a higher throughput than programs that use the Java Kafka client directly (see section Performance below).

Production Ready CI Badge Sonatype Releases Sonatype Snapshots javadoc ZIO Kafka Scala Steward badge

Introduction

Apache Kafka is a distributed event streaming platform that acts as a distributed publish-subscribe messaging system. It enables us to build distributed streaming data pipelines and event-driven applications.

Kafka has a mature Java client for producing and consuming events, but it has a low-level API. Zio-kafka is a ZIO native client for Apache Kafka. It has a high-level streaming API on top of the Java client. So we can produce and consume events using the declarative concurrency model of zio-streams. In addition, zio-kafka supports an even higher level API where you only write the processing part and the rest is handled by zio-kafka.

Features

  • Exposes the Java Kafka consumer, producer and admin clients with a ZIO based interface.
  • Consuming (see Consumer):
  • Producing (see Producer):
    • 2 APIs: streaming and ZIO workflow based
    • supports custom serialization
    • allows for batches for highest throughput
    • optionally await broker acknowledgements
    • supports transactions
    • optional retries after authentication/authorization errors
    • exposes metrics
    • diagnostics API
  • Admin API:
    • exposes most of the admin client methods with a ZIO based interface
    • safer than the wrapped java client by waiting for the operation to be done
  • Proper errors when broker expects SSL (no OOM crashes), see Security
  • Test kit with embedded kafka broker
  • Well documented
  • Community support via Discord
  • Commercial support via Ziverge

Installation

To use zio-kafka, add the following lines to your build.sbt file:

libraryDependencies += "dev.zio" %% "zio-kafka"         % "4.2.0"
libraryDependencies += "dev.zio" %% "zio-kafka-testkit" % "4.2.0" % Test

The Getting Started page walks through running a local Kafka broker and writing a first producer and consumer. For a longer tour of the different ways you can use zio-kafka, read the zio-kafka tutorial.

Documentation

If you are reading this page from https://zio.dev/zio-kafka/, the documentation is in the sub-pages and can be found by scrolling the left panel of this page.

If you are reading this from GitHub, the documentation can be found in the docs folder.

Resources

The following articles, videos and example projects show zio-kafka in practice.

Articles

Video

Example Projects

  • Kafka BigQuery Express by Adevinta (November 2024) A production system to copy data from Kafka to BigQuery, safely and cost-effectively. (See also the video "Optimizing Data Transfer...".)
  • zio-kafka-showcase by Jorge Vásquez, Example project that demonstrates how to build Kafka based microservices with Scala and ZIO
  • zio-kafka-demo1 (December 2022), example consumer and producer using zio-kafka 2.0.5
  • zio-kafka-example-app by Ziverge (December 2020), example application using zio-kafka 0.8.0

Adopters

Here is a partial list of companies using zio-kafka in production.

Want to see your company here? Submit a PR!

Performance

Often, zio-kafka programs consume with a higher throughput than programs that use the java-kafka client directly. Read on for the details.

Zio-kafka makes it easy to process partitions in parallel: Consumer.consumeWith does so by default, and with the streaming API you can use Consumer#partitionedStream with ZStream#flatMapPar. The default java-kafka client does not provide parallel processing. Of course, there is some overhead in buffering records and distributing them to the fibers that need them. On 2024-11-23, we estimated that zio-kafka consumes faster than the java-kafka client when processing takes more than ~1.2ms per 1000 records. The precise time depends on many factors. Please see this article for more details.

If you do not care for the convenient ZStream based API that zio-kafka brings, and latency is of absolute importance, using the java based Kafka client directly is still the better choice.

Developers

Documentation

Learn more on the ZIO Kafka homepage!

Contributing

For the general guidelines, see ZIO contributor's guide.

Code of Conduct

See the Code of Conduct

Support

Come chat with us on Badge-Discord.

Credits

This library is heavily inspired and made possible by the research and implementation done in Alpakka Kafka, a library maintained by the Akka team and originally written as Reactive Kafka by SoftwareMill.

License

License

Copyright 2021-2026 Itamar Ravid and the zio-kafka contributors.

About

A fast Kafka client for ZIO and ZIO Streams

Resources

Security policy

Stars

365 stars

Watchers

15 watching

Forks

Releases

Used by

Contributors

Languages