Skip to content

Tags

Tags

v1.2.0

Toggle v1.2.0's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
chore: prepare release v1.2.0 (#613)

* chore: prepare release v1.2.0

Require Kayrock 1.1.0 (the Kafka 4.4.0 protocol structs) and finalize the
1.2.0 changelog. Document the protocol-version and error-atom changes in
UPGRADING.md.

The lock entry still points at the Kayrock git master pin: run mix deps.get
after Kayrock 1.1.0 is published to Hex.

* chore: lock kayrock 1.1.0

Kayrock 1.1.0 is on Hex now, so replace the git master pin in the lock
with the published Hex package.

v1.1.3

Toggle v1.1.3's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
chore: prepare release v1.1.3 (#608)

v1.1.2

Toggle v1.1.2's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
chore: release v1.1.2 (#597)

Finalise the 1.1.2 CHANGELOG section and bump the version. Ships the
produce acks fix (#593), leaderless-partition metadata retention (#595)
and consumer fetch retry (#596).

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

v1.0.3

Toggle v1.0.3's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
fix(produce, consumer): honour acks, and survive leader moves & SSL b…

…roker restarts (1.0.x) (#591)

* fix(produce): honour the acks option and make acks=0 fire-and-forget work

0.x translated `required_acks` onto the wire in the legacy adapter; the 1.0
rewrite removed that adapter and the translation with it, while
`KafkaEx.API.produce/5` kept documenting the option. The request builder reads
`:acks`, so since 1.0 every produce went out with `acks: -1` regardless of what
the caller asked for.

The option is now resolved once in the client (`:acks` > deprecated
`:required_acks` > -1) and feeds both the wire and the produce telemetry, so the
two can no longer disagree. Values other than -1/0/1 are rejected with
`{:error, :invalid_acks}` instead of reaching Kayrock's int16 encoder, where
`:all` raised inside the client and 65_536 silently truncated to 0 while the
client waited for a response that never came.

`acks: 0` was unreachable and, once reachable, crashed on `byte_size/1` over the
async send result. It now takes a dedicated path that skips the retry layer (a
resend would duplicate records, since the broker sends nothing to confirm the
first one) and returns `%RecordMetadata{base_offset: nil}`, which widens the
typespec to `non_neg_integer() | nil`.

Two failure modes that fire-and-forget makes routine are fixed alongside it: a
failed async send now closes the broken socket the way the sync send already
did, and `KafkaEx.Client` grows `{:tcp_error, _, _}` / `{:ssl_error, _, _}` plus
a catch-all `handle_info/2` — defining any clause replaces the one `use
GenServer` injects, so a connection reset (how a broker reports a rejected
acks=0 batch) killed the client along with every socket and all metadata.

* fix(produce): act on code review of the acks fix

Log unmatched client messages at warning level. The catch-all was added so an
unexpected message no longer kills the client, but debug level put it below the
default threshold — a late broker response arriving on an active socket was
being discarded with no trace at all.

Stop the acks=0 integration tests from racing partition leader election. Before
this branch, required_acks: 0 silently meant acks: -1, so those produces were
synchronous and retried; now they are genuinely fire-and-forget and a batch sent
before the leader is elected is dropped without an error. Establish the leader
with an acknowledged produce first, and poll for the offset instead of sleeping
a fixed 500 ms.

* chore: release v1.0.3 — the acks option reaches the broker again

Backports the produce acks fix from master onto the 1.0.x line.

* fix(telemetry): keep the connection close reason a known atom

A socket error handed :ssl_error's raw term to the close event, which the
contract fixes to a small set of atoms — an alert tuple there breaks handlers
that match known reasons and inflates metric cardinality.

Also covers the send_async_request/2 error branch, which was untested.

* fix(consumer): retry transient fetch errors instead of stopping

GenConsumer stopped on any fetch error other than :offset_out_of_range,
including ones Retry.transient_error?/1 already classifies as transient. With
ConsumerGroup supervising at max_restarts: 0, a single partition losing its
leader took the whole group down. RF=1 makes the window wide enough to hit
reliably, but the same path runs on any leader move.

Retries are unbounded with jittered exponential backoff, as in brod, KafkaJS
and librdkafka. The backoff is held as a deadline rather than a bare GenServer
timeout, because any inbound message re-arms that callback with timeout 0 and
would otherwise cut the wait short.

Backport of the master fix. Brings Retry.backoff_delay_jittered/3 with it, which
this line did not have.

* fix(client): log why partition leader lookup failed

:no_broker collapses :no_such_topic, :no_such_partition and :broker_not_found
into one atom, so a leaderless partition reads the same as a deleted topic. The
reason now reaches the log with the topic and partition; the returned error is
unchanged, so this is safe for a patch release.

* fix(consumer): act on code review of the fetch retry fix

Three changes, all from review of the commits above.

Retry.backoff_delay/3 computed base * :math.pow(2, attempt) before applying the
cap. Every previous caller was bounded at 3-5 attempts; the new fetch loop is
not, and the float overflows into ArithmeticError past attempt 1023 - roughly 85
minutes of continuous failure at the 5s cap. It would have killed the consumer,
and with max_restarts: 0 the group, which is what this work exists to prevent.
Now an exact integer shift with a clamped exponent.

handle_offset_out_of_range/1 hard-matched {:ok, offset} from
latest_offset/earliest_offset. Those need a live leader too, so a leader move
during an offset reset raised MatchError. The failure now feeds the same backoff.
The startup path (load_offsets/1) still raises by design; documented as such.

The retry warning is throttled after the backoff reaches its cap.

* fix(consumer): survive SSL transport errors and flag stuck partitions

Backport of the code-review follow-ups from master onto 1.0.x.

- A non-atom transport reason (an SSL {:tls_alert, _} tuple) was normalised to
  :unknown, which the fetch loop treats as fatal — so on TLS clusters a broker
  restart still took the whole group down, defeating the retry fix. Such reasons
  now normalise to a retryable :transport_error.
- Retries stay unbounded (brod/Java/librdkafka do the same), but after
  :fetch_unavailable_warn_ms (default 30s) of continuous failure a partition is
  surfaced once via Logger.error and a [:kafka_ex, :consumer, :partition_unavailable]
  telemetry event. Mirrors librdkafka's topic.metadata.propagation.max.ms.
- An acks=0 async send failure emitted a :send_error close event while the
  active-mode {:tcp_error} handler emitted a second :recv_error for the same
  connection death; the network layer no longer emits, leaving the handler the
  single owner.
- The per-lookup "no broker after refresh" log drops to debug.
- CHANGELOG marks the acks=0 durability change as BREAKING on this patch line.

* style(consumer): trim the fetch-resilience comments to one-line whys

Comment-only change: cut the multi-line rationale blocks added with the SSL
transport-error / partition-unavailable work down to a single non-obvious "why"
each (or nothing where the code is self-evident). No logic changes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(metadata): keep leaderless partitions in cluster metadata (1.0.x)

Backports the metadata-retention half of the leader-move fix that the
consumer fetch retry depends on. Parse keeps partitions carrying a
leader-move error (leader -1) instead of dropping them, the partitioner
counts them so a key keeps its partition, and select_node reports
:leader_not_available. Without this the retry had nothing to recover:
a leaderless partition dropped out of the cluster model entirely.

Also aligns the fetch retry base delay to 250ms, matching master.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

v1.1.1

Toggle v1.1.1's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
Release/v1.1.1 (#588)

* feat(client): add tracked_topics + metadata-missing state fields

* feat(client): add pure MetadataLog decision helpers

* fix(client): refresh metadata for used topics only, not the whole catalog

* fix(client): merge partial metadata on give-up and log missing topics edge-triggered

* test(client): cover metadata give-up merge and edge-triggered missing-topic logging

* refactor(client): align MetadataLog topic-name spec with codebase convention

* test(client): integration test - unrelated topic deletion does not storm logs

* chore: release v1.1.1 — metadata tracks only used topics; storm-free logging

* refactor(client): trim metadata comments to one-line whys; add missing-set-change test

* test(client): prove metadata narrowing = latency-not-loss (unit + lifecycle integration) + consumer-group topic-deletion resilience

* fix(test): make lifecycle metadata_refresh_test async: false (CaptureLog is VM-wide)

* test(client): make metadata_missing_test hermetic by advertising the setup socket's ephemeral port (#589)

The setup binds a real socket on an ephemeral port via :gen_tcp.listen(0),
but build_v0_metadata_response hard-coded the advertised broker port as 9092.
Because ClusterMetadata.merge_brokers/2 only preserves a live socket when both
host and port match, the port mismatch caused the client to drop the stubbed
socket and attempt a real, unstubbed reconnect to localhost:9092 on the second
update_metadata call. On a clean host (CI) that reconnect fails with
:econnrefused, producing 3 flaky failures; it only passed locally when
something happened to be listening on 9092.

Thread the setup socket's real ephemeral port through setup ->
stub_metadata_response -> build_v0_metadata_response and advertise it instead
of 9092, so the test no longer depends on the network.

* test(client): make metadata_narrowing_test hermetic by threading the setup socket's ephemeral port

Same root cause as #589 but in the narrowing test: build_v0_metadata_response
hard-coded the advertised broker port as 9092, so merge_brokers/2 dropped the
stubbed socket (host+port must both match) and the second update_metadata call
attempted a real reconnect to localhost:9092 -> :econnrefused on CI. Thread the
setup socket's ephemeral port through instead of 9092 so the test is network-free.

* fix(client): stop re-running the metadata retry-sleep for already-known-missing topics

A tracked topic that is permanently missing (e.g. deleted) was driving the full
retrieve_metadata retry ladder — @retry_count network attempts with a 300ms sleep
between each (~600ms) inside the client GenServer — on every periodic refresh,
head-of-line-blocking all requests on that client every ~30s.

Spend the retry-sleep budget only on newly-missing topics; a topic already in
metadata_missing goes straight to the merge/give-up branch. Transient-gap
tolerance on first miss is preserved; the recurring stall is gone.

* docs(changelog): note the deleted-but-tracked-topic stall fix and the tracked-topic pruning limitation

* docs(changelog): tighten the metadata-fix entries

* feat(consumer): seed a GenConsumer's own client with initial_topics: [topic]

When a GenConsumer starts its own client (no shared :client passed), the client
started with an empty tracked set, so its first metadata refresh was scoped to
the whole cluster catalog. In a group with N partition-consumers that meant N
simultaneous whole-catalog fetches at startup/rebalance. Seed initial_topics so
each client's first refresh is scoped to just its topic.

Also broadens the ssl_options typespec to cover any Erlang :ssl option (it only
listed four keys but Config.ssl_options/0 passes :verify etc.), surfaced by
type-checking the Client.start_link stub in the new test.

* docs(changelog): note the consumer client initial_topics scoping

* refactor(client): review cleanups — boolean newly-missing check, drop dead MetadataLog.heartbeat_ms/0, honest refresh comment

Post-review polish (no behavior change):
- parse_metadata_response: Enum.any? boolean instead of building a rejected list
- MetadataLog: remove the unused heartbeat_ms/0 accessor (@heartbeat_ms stays)
- update_metadata: comment no longer claims 'never the whole cluster catalog'
  (an empty tracked set still fetches all until the [] vs nil builder fix)

---------

Co-authored-by: Àbéjídé Àyodélé <abejideayodele@gmail.com>

v1.0.2

Toggle v1.0.2's commit message

Verified

This commit was signed with the committer’s verified signature.
Argonus Piotr Rybarczyk
refactor(client): review cleanups — boolean newly-missing check, drop…

… dead MetadataLog.heartbeat_ms/0, honest refresh comment

Post-review polish (no behavior change):
- parse_metadata_response: Enum.any? boolean instead of building a rejected list
- MetadataLog: remove the unused heartbeat_ms/0 accessor (@heartbeat_ms stays)
- update_metadata: comment no longer claims 'never the whole cluster catalog'
  (an empty tracked set still fetches all until the [] vs nil builder fix)

v1.1.0

Toggle v1.1.0's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
chore: release v1.1.0 (#576)

* chore: release v1.1.0

Bump @Version to 1.1.0 and finalize the CHANGELOG: retitle Unreleased →
1.1.0 (2026-07-16), add the missing ListGroups admin API (#565) entry, and drop
the stale "heartbeat" mention from the :request_timeout bullet (heartbeat derives
its own deadline).

* fix: retry data-plane requests on :econnreset / :not_connected (v1.0.x parity)

The v1.1.0 retry-classification rework narrowed data-plane retriability to
transient_error? ∪ leadership_error?; :econnreset and :not_connected fell
outside it and began failing fast, where v1.0.x retried all transport errors
and self-healed (the client retry loop reconnects the broker on the next
attempt). Classify both as transient again.

* docs: finalize 1.1.0 CHANGELOG (release date + data-plane retry wording)

Set the 1.1.0 date to the release day and fold the transient-connection-drop
clarification (:closed / :econnreset / :not_connected / :timeout still retry)
into the existing 'data-plane fail fast on fatal errors' Changed entry, instead
of a standalone Fixed bullet that implied a released regression (the retry
narrowing never shipped — it was introduced and refined within 1.1.0).

v1.0.1

Toggle v1.0.1's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
chore: add AGENTS.md as canonical agent guide; CLAUDE.md symlinks to …

…it (#558)

Establish AGENTS.md (the cross-tool convention read by Claude Code, Cursor,
Copilot, etc.) as the single source of agent guidance for this repo:

- Add ash-project/usage_rules (dev-only) and a `usage_rules` mix.exs config
  that inlines Kayrock's usage rules into AGENTS.md under a managed marker
  block. KafkaEx is built on Kayrock, so its protocol rules are the relevant
  dependency rules. Refresh with `mix usage_rules.sync`.
- Hand-written top section carries the former CLAUDE.md project guidance
  (architecture, commands, conventions) plus a pointer to usage-rules.md for
  public API call patterns.
- CLAUDE.md becomes a symlink to AGENTS.md so Claude Code keeps reading it.

v1.0.0

Toggle v1.0.0's commit message

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
chore: clean up IO.inspect in delete_topics docstring example (#532)

Replace IO.inspect with IO.puts/inspect in the @doc example for
KafkaEx.API.delete_topics/3,4 so the rendered hexdocs do not model an
anti-pattern.

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

v0.15.0

Toggle v0.15.0's commit message
Release v0.15.0