kcl

• Introduction
• Getting Started / Installation
• Configuration
• Autocompletion
• Group Consuming
• Share Groups
• Schema Registry
• Local Fake Cluster
• API at a Glance
• Examples

kcl is a complete, pure Go command line Kafka client. Think of it as your one stop shop to do anything you want to do with Kafka -- producing, consuming, administering, transactions, ACLs, share groups, and so on.

Unlike the small size of kcat (formerly kafkacat), this binary is ~15M compiled. It is, however, still fast, has rich consuming and producing formatting options, and a complete Kafka administration interface that tracks the upstream protocol closely.

If you have a Go installation:

go install github.com/twmb/kcl@latest

This installs kcl from the latest release. You can optionally suffix with @v#.#.# to install a specific version. When installed this way, kcl automatically reports itself to brokers as kcl/<version> via the Kafka protocol's client ID (useful for ACL audit logs and broker-side metrics). kcl version prints the version and build details of the binary you have.

Otherwise, download a release from the releases page.

A container image is also published for each release, useful as a short-lived Kubernetes or CI client:

kubectl run kafka-client --rm -i --restart=Never \
  --image=ghcr.io/twmb/kcl:latest \
  -- --no-config-file -B kafka-headless:9092 topic list

(the image's entrypoint is already kcl, so args after -- go straight to it; suffix the image with :v#.#.# for a specific release)

The image has no shell, so it cannot serve as a CI job's container; run it from a step with docker run or kubectl run.

kcl is usable out of the box against localhost:9092; no config is required for the common case of probing a local cluster. For real clusters you can either use flags, environment variables, or a config file. The config file supports multiple named profiles so that switching between clusters is easy.

Priority (highest wins):

1. -B/--bootstrap-servers (seed brokers only)
2. -X key=value flags (repeatable; any config key)
3. KCL_<KEY> environment variables
4. Active profile in the config file (--profile/-C, KCL_PROFILE, or current_profile)
5. Top-level config file keys (flat layout)
6. Built-in defaults

By default, kcl reads its config from your OS user-config directory, typically ~/.config/kcl/config.toml. The default path can be overridden with --config-path or KCL_CONFIG_PATH.

The configuration supports TLS, SASL (PLAIN, SCRAM-SHA-256, SCRAM-SHA-512, AWS_MSK_IAM), seed brokers, and client/server timeouts. Timeouts accept Go duration strings (500ms, 5s, 2m30s).

kcl -X help describes every key; kcl -X list names them.

kcl profile create writes a profile from the same flags a one-off command takes:

kcl profile create prod -B kafka-prod-1:9092,kafka-prod-2:9092
kcl profile create cicd -B kafka-staging:9092 -X dial_timeout=2s -X sasl.mechanism=plain -X sasl.user=ci -X sasl.pass=secret

If nothing is current yet, the new profile is; otherwise kcl profile use NAME. A value may reference an environment variable as ${NAME}, so a password can live in your environment rather than in the file. kcl profile set applies the same flags to the current profile, or to the one -C names.

# ~/.config/kcl/config.toml
current_profile = "prod"

[profiles.prod]
seed_brokers   = ["kafka-prod-1:9092", "kafka-prod-2:9092"]
broker_timeout = "10s"

[profiles.cicd]
seed_brokers   = ["kafka-staging:9092"]
dial_timeout   = "2s"       # fail fast in CI
broker_timeout = "5s"
retry_timeout  = "5s"

[profiles.local]
seed_brokers = ["localhost:9092"]

Then:

kcl topic list                  # uses "prod"
kcl -C cicd topic list          # switches to "cicd" for one command
kcl -B other-host:9092 topic list   # one-off override of seed brokers

Thanks to cobra, autocompletion exists for bash, zsh, and powershell.

Bash example to put in .bashrc:

if [ -f /etc/bash_completion ] && ! shopt -oq posix; then
    . /etc/bash_completion
    . <(kcl misc gen-autocomplete -kbash)
fi

Group consuming is supported with the -g/--group flag on kcl consume. The default balancer is cooperative-sticky (incremental rebalancing, Kafka 2.4+), which is incompatible with the older eager balancers (roundrobin, range, sticky). If your existing group has members using eager balancing, pass --balancer explicitly.

kcl group describe shows per-partition committed offsets, lag, and member assignments. kcl group seek resets committed offsets via AlterOffsets; kcl group offset-delete deletes specific partitions' committed offsets.

Share groups (KIP-932, Kafka 4.0+) are supported via kcl consume --share-group NAME. The --share-ack-type flag controls how each fetched record is acknowledged:

• accept (default) -- mark the record as successfully processed.
• release -- put the record back into the pool for redelivery (bumps delivery count). Useful for peeking at records without consuming them.
• reject -- archive the record as unprocessable (bumps delivery count, no redelivery). Useful for force-draining or exercising DLQ-style flows.

kcl share-group has its own list, describe, seek, delete, and offset-delete subcommands.

The Schema Registry is a separate HTTP service from the Kafka brokers. Point kcl at it with -R/--registry (comma-separated URLs), -X registry.urls=..., or a registry config section; it defaults to http://localhost:8081 (mirroring the localhost:9092 broker default). Auth is optional: basic auth (registry.user / registry.pass), a bearer token (registry.bearer_token), and TLS for https URLs (registry.tls.*, mirroring the Kafka tls.* keys).

kcl registry administers the registry as a thin layer over franz-go's typed pkg/sr API:

kcl registry subject list                      # list subjects
kcl registry schema list mytopic-value         # list a subject's versions
kcl registry schema list                       # all schemas across all subjects
kcl registry schema create mytopic-value -s schema.avsc   # register (avro by default)
kcl registry schema get -S mytopic-value       # latest schema for a subject
kcl registry schema get --id 5                 # schema by global id
kcl registry schema references mytopic-value   # schemas that reference this one
kcl registry schema delete mytopic-value -v 3  # soft-delete a version (--permanent to hard)
kcl registry subject delete mytopic-value      # soft-delete a whole subject
kcl registry compatibility set BACKWARD mytopic-value   # set per-subject compatibility
kcl registry compatibility get                 # global compatibility
kcl registry schema check-compatibility mytopic-value -s new.avsc --verbose   # check a candidate schema
kcl registry mode get                          # global mode
kcl registry context list                      # list contexts (namespaces)
kcl registry --context myctx subject list      # scope to a context

(schema create is also available as schema register.)

kcl produce and kcl consume can transcode between JSON and the Schema Registry binary wire format (the magic byte + 4-byte schema id, plus the Protobuf message-index). Avro, JSON Schema, and Protobuf are supported. Encoding is the inverse of decoding: produce reads JSON and writes schema-encoded bytes; consume reads schema-encoded bytes and writes JSON, which then flows through the normal %v/%k format verbs.

On produce, --schema (value) and --key-schema (key) take a small spec that resolves an existing schema (producing never registers — use kcl registry schema create for that):

topic[@VERSION]          schema for <topic>-value / <topic>-key (latest)
NAME[@VERSION]           a subject (bare), e.g. orders-value, orders-value@3
subject:NAME[@VERSION]   an explicit subject (escape hatch for odd names)
id:N                     a registered schema id

VERSION is a number or latest (default); any form may add a trailing #MESSAGE to pick the protobuf message in a multi-message schema.

# Encode values with the latest registered <topic>-value schema:
echo '{"id":"a","n":1}' | kcl produce orders --schema topic

# By explicit subject/version, or by id; encode the key too:
kcl produce orders --schema orders-value@3
kcl produce orders -f '%k %v\n' --key-schema id:7 --schema id:8

# Protobuf, selecting the message:
kcl produce orders --schema topic#com.acme.Order

On consume, opt in with --decode (both key and value), or --decode=value / --decode=key for one. The schema id is read from each record and the schema is fetched and cached automatically; records not in the wire format are printed unchanged:

kcl consume orders --decode
kcl consume orders --decode=value -f '%k -> %v\n'

The subject defaults to the TopicNameStrategy (<topic>-value / <topic>-key) when not given explicitly. For an end-to-end playground with no external dependencies, kcl fake --seed-demo stands up a registry and seeds schema-encoded topics (see below).

To test against a real registry instead of the fake, run one in Docker, e.g. Redpanda (registry on 8081):

docker run -d --name redpanda -p 9092:9092 -p 8081:8081 \
  redpandadata/redpanda redpanda start --schema-registry-addr 0.0.0.0:8081 \
  --kafka-addr 0.0.0.0:9092 --advertise-kafka-addr localhost:9092
kcl -R http://localhost:8081 registry subject list

kcl fake runs a kfake cluster in-process and prints the listen addresses. Point any Kafka client (including another kcl invocation) at those addresses; SIGINT or SIGTERM exits cleanly. State is in-memory by default; pass --data-dir PATH to persist, and --sync for fsync-on-write durability.

This is NOT a production broker. kfake implements the user-facing Kafka protocol surface (produce, fetch, groups, transactions, ACLs, share groups) but intentionally omits broker-to-broker / KRaft-internal requests and is not performance-tuned. It's great for probing, learning, integration tests, CI pipelines, and demos without Docker.

kcl fake                                 # 3 brokers on 9092,9093,9094
kcl fake --ports 9092,9093,9094          # 3 brokers on specific ports
kcl fake --ports 9092                    # single-broker cluster
kcl fake -d /tmp/kfake --sync            # persistent, durable
kcl fake --seed-topic foo:10,bar:3       # pre-create topics
kcl fake --as-version 3.9                # cap advertised API versions
kcl fake --acls --sasl 'plain:$USER:$PW' # SASL superuser from env vars
kcl fake -c group.consumer.heartbeat.interval.ms=500  # broker config
kcl fake -l debug                        # verbose kfake logs
kcl fake --registry=false                # disable the bundled schema registry
kcl fake --seed-demo                     # seed schema-encoded + plain demo topics

kcl fake also serves an in-memory Schema Registry (srfake) on port 8081 by default, so the same process backs schema-aware produce/consume; disable it with --registry=false or move it with --registry-port. If the port is busy (e.g. a real registry is already running) kcl warns and continues without it, unless --registry/--registry-port was passed explicitly.

--seed-demo creates demo-avro, demo-proto, and demo-json (each with a registered schema of that type) plus demo-plain (no schema), all sharing the same {id, count} shape, and produces a few records to each -- a one-command, Docker-free playground for the whole schema-registry flow:

kcl fake --seed-demo
kcl consume demo-avro -o start --decode   # decodes back to JSON
kcl consume demo-plain -o start           # plain JSON, no schema

The --sasl flag accepts MECHANISM:USER:PASS (repeatable). Supported mechanisms: plain, scram-sha-256, scram-sha-512. User and password go through os.ExpandEnv, so quoting the argument keeps the shell from expanding first and lets the broker name env vars pick up the secrets.

The best way to explore kcl is kcl --help and then kcl <cmd> --help. The top-level commands are:

kcl
 acl            -- list/create/delete ACLs
 client-metrics -- manage client telemetry subscriptions (KIP-714)
 cluster        -- metadata, quorum, feature flags, leader elections, KRaft voters
 config         -- alter/describe topic, broker, group, client-metrics configs
 consume        -- consume records (classic group, share group, or direct)
 dtoken         -- delegation token commands
 fake           -- start a local in-process kfake cluster for testing
 group          -- classic / KIP-848 consumer group operations
 logdirs        -- per-partition log directory operations
 misc           -- api-versions, raw-req, error lookups, completion
 produce        -- produce records
 profile        -- manage connection profiles / config
 quota          -- alter/describe/resolve client quotas
 reassign       -- alter/list/verify partition reassignments
 registry       -- schema registry: schemas, subjects, compatibility, mode
 share-group    -- share group operations (KIP-932)
 topic          -- list/create/describe/delete/add-partitions/trim-prefix/list-offsets
 txn            -- describe active transactions / producers
 user           -- SCRAM user credential management

Output format for every command is controlled by the global --format flag (text, json, or awk). JSON output is stable ({_command, _version, ...} envelope) and suitable for piping into jq. Text output is tab-aligned and may change between releases; awk output is headerless TSV with a stable column order, and an empty or unknown cell prints as - so every row has every field. To learn a command's columns, run it with --format awk-header: it prints the header row for the flags you gave and exits without connecting. Column names are hyphen-delimited (TOPIC-ID, LEADER-EPOCH, etc.). Every row that reports a per-item result ends in ERROR and MESSAGE: the Kafka error name and the text the broker attached, both empty on success. Note that consume and produce deliberately repurpose --format as the per-record format string (records don't fit the table envelope).

For tooling and agents that want to introspect kcl's entire command tree programmatically (names, flags, examples, help text), use the global --help-json flag at the root:

kcl --help-json | jq '.commands[] | .name'

Interactive confirmation prompts on destructive commands (group seek, share-group seek, topic trim-prefix, config alter, acl delete) are skipped with --yes/-y. When stdin is not a terminal the prompt answers no and the command prints the plan it would have carried out and exits 0, so < /dev/null is a scriptable dry run. Commands with --dry-run/-d mark the document with "dry_run":true.

Consume topic foo, print values:

kcl consume foo

Advanced formatting -- key, value, and headers:

kcl consume foo -f "KEY=%k, VALUE=%v, HEADERS=%{%h{ '%k'='%v' }}\n"

Group consuming from topics foo and bar:

kcl consume -g mygroup foo bar

Share group consuming (Kafka 4.0+), peeking at records without consuming:

kcl consume --share-group sg1 --share-ack-type release foo

From a specific timestamp:

kcl consume foo -o @2024-01-15
kcl consume foo -o @-1h            # 1 hour ago
kcl consume foo -o @-30m:@now      # 30 minutes ago to now

Newline-delimited value to topic foo:

echo fubar | kcl produce foo

Values from a file:

kcl produce foo < baz

Produce key k and value v from a single line:

echo "key: k, value: v" | kcl produce foo -f 'key: %k, value: %v\n'

Produce with headers:

echo "k v 2 h1 v1 h2 v2" | kcl produce foo -f '%k %v %H %h{%k %v }\n'

kcl topic create foo                              # uses cluster default partitions/replication
kcl topic create foo -p 6 -r 3                    # 6 partitions, 3 replicas
kcl topic describe foo                            # partitions, configs, health
kcl topic describe --topic-id <uuid>              # lookup by UUID (KIP-516)
kcl cluster metadata                              # broker list, controller
kcl cluster describe                              # the DescribeCluster view
kcl topic list-offsets foo --at -1h               # the offset an hour ago
kcl cluster features describe                     # feature flags (KIP-584)
kcl cluster features update share.version=1 --upgrade-type safe-downgrade
kcl cluster features update --release-version 4.4 --dry-run   # every feature to its 4.4 level
kcl group list                                    # classic + KIP-848 + share groups
kcl group describe mygroup
kcl group seek mygroup --to end --yes
kcl acl list

# One-command playground (brokers + registry + seeded schema topics):
kcl fake --seed-demo
kcl consume demo-avro -o start --decode

# Register a schema and round-trip JSON <-> Avro binary:
kcl registry schema register foo-value -s user.avsc
echo '{"id":"a","n":1}' | kcl produce foo --schema topic
kcl consume foo -o start --decode

# Inspect the registry:
kcl registry subject list
kcl registry schema get -S foo-value
kcl registry compatibility get foo-value

Start a fake in one shell, use it from another:

# shell 1
kcl fake --seed-topic foo:3

# shell 2 (fake prints 127.0.0.1:<port> -- pick any)
kcl -B 127.0.0.1:<port> topic list
seq 1 5 | kcl -B 127.0.0.1:<port> produce foo
kcl -B 127.0.0.1:<port> consume foo -n 5 -o start

Or set a persistent profile for the fake so -B isn't needed on each invocation:

kcl profile create fake -B 127.0.0.1:<port>
kcl -C fake topic list

Commands exit 0 on success, 1 on any Kafka-level or per-item failure (e.g. deleting one topic out of three, where one doesn't exist, exits 1), and 2 on a usage error. Under --format json stdout is always one valid JSON document, an error included: a failed command prints {_command, _version, code, error}. Under text and awk, errors go to stderr. This makes kcl safe to script against.