kplay (short for "kafka-playground") lets you inspect messages in a Kafka topic in a simple and deliberate manner. Using it, you can pull one or more messages on demand, decode them based on a configured encoding format, peruse them in a list, persist them to your local filesystem, or forward them to S3.

homebrew:

brew install dhth/tap/kplay

go:

go install github.com/dhth/kplay@latest

Or get the binaries directly from a release. Read more about verifying the authenticity of released artifacts here.

kplay offers 4 commands:

• tui: browse messages in a kafka topic via a TUI
• serve: browse messages in a kafka topic via a web interface
• scan: scan a topic for messages, and optionally save them to your local filesystem
• forward: consume messages from a topic, and forward them to a remote destination

This will start a TUI which will let you browse messages on demand. You can then browse the message metadata and value in a pager. By default, kplay will consume messages from the earliest possible offset, but you can modify this behaviour by either providing an offset or a timestamp to start consuming messages from.

Usage:
  kplay tui <PROFILE> [flags]

Flags:
  -o, --from-offset string      start consuming messages from this offset; provide a single offset for all partitions (eg. 1000) or specify offsets per partition (e.g., '0:1000,2:1500')
  -t, --from-timestamp string   start consuming messages from this timestamp (in RFC3339 format, e.g., 2006-01-02T15:04:05Z07:00)
  -h, --help                    help for tui
  -O, --output-dir string       directory to persist messages in (default "$HOME/.kplay")
  -p, --persist-messages        whether to start the TUI with the setting "persist messages" ON
  -s, --skip-messages           whether to start the TUI with the setting "skip messages" ON

Global Flags:
  -c, --config-path string   location of kplay's config file (can also be provided via $KPLAY_CONFIG_PATH)
      --debug                whether to only display config picked up by kplay without running it

This will start kplay's web interface which will let you browse messages on demand. By default, kplay will consume messages from the earliest possible offset, but you can modify this behaviour by either providing an offset or a timestamp to start consuming messages from.

Usage:
  kplay serve <PROFILE> [flags]

Flags:
  -o, --from-offset string      start consuming messages from this offset; provide a single offset for all partitions (eg. 1000) or specify offsets per partition (e.g., '0:1000,2:1500')
  -t, --from-timestamp string   start consuming messages from this timestamp (in RFC3339 format, e.g., 2006-01-02T15:04:05Z07:00)
  -h, --help                    help for serve
  -O, --open                    whether to open web interface in browser automatically
  -S, --select-on-hover         whether to start the web interface with the setting "select on hover" ON

Global Flags:
  -c, --config-path string   location of kplay's config file (can also be provided via $KPLAY_CONFIG_PATH)
      --debug                whether to only display config picked up by kplay without running it

This command is useful when you want to view a summary of messages in a Kafka topic (ie, the partition, offset, timestamp, and key of each message), and optionally save the message values to your local filesystem.

Usage:
  kplay scan <PROFILE> [flags]

Flags:
  -b, --batch-size uint         number of messages to fetch per batch (must be greater than 0) (default 100)
  -d, --decode                  whether to decode message values (false is equivalent to 'encodingFormat: raw' in kplay's config) (default true)
  -o, --from-offset string      scan messages from this offset; provide a single offset for all partitions (eg. 1000) or specify offsets per partition (e.g., '0:1000,2:1500')
  -t, --from-timestamp string   scan messages from this timestamp (in RFC3339 format, e.g., 2006-01-02T15:04:05Z07:00)
  -h, --help                    help for scan
  -k, --key-regex string        regex to filter message keys by
  -n, --num-records uint        maximum number of messages to scan (default 1000)
  -O, --output-dir string       directory to save scan results in (default "$HOME/.kplay")
  -s, --save-messages           whether to save kafka messages to the local filesystem

Global Flags:
  -c, --config-path string   location of kplay's config file (can also be provided via $KPLAY_CONFIG_PATH)
      --debug                whether to only display config picked up by kplay without running it

This command is useful when you want to consume messages in a kafka topic as part of a consumer group, decode them, and forward the decoded contents to a remote destination (AWS S3 is the only supported destination for now).

This command is intended to be run in a long running containerised environment; as such, it accepts configuration via the following environment variables.

If needed, this command can also start an HTTP server which can be used for health checks (at /health).

Usage:
  kplay forward <PROFILE>,<PROFILE>,... <DESTINATION> [flags]

Examples:
kplay forward profile-1,profile-2 arn:aws:s3:::bucket-to-forward-messages-to/prefix

Flags:
  -h, --help   help for forward

Global Flags:
  -c, --config-path string   location of kplay's config file (can also be provided via $KPLAY_CONFIG_PATH)
      --debug                whether to only display config picked up by kplay without running it

kplay's configuration file looks like the following:

profiles:
  - name: json-encoded
    authentication: none
    encodingFormat: json
    brokers:
      - 127.0.0.1:9092
    topic: kplay-test-1

  - name: proto-encoded
    authentication: aws_msk_iam
    encodingFormat: protobuf
    protoConfig:
      descriptorSetFile: path/to/descriptor/set/file.pb
      descriptorName: sample.DescriptorName
    brokers:
      - 127.0.0.1:9092
    topic: kplay-test-2

  - name: raw
    authentication: none
    encodingFormat: raw
    brokers:
      - 127.0.0.1:9092
    topic: kplay-test-3

kplay supports decoding messages that are encoded in two data formats: JSON and protobuf. It also supports handling the message bytes as raw data (using the encodingFormat "raw").

For decoding protobuf encoded messages, kplay needs to be provided with a FileDescriptorSet and a descriptor name. Consider a .proto file like the following:

// application_state.proto
syntax = "proto3";

package sample;

message ApplicationState {
  string id = 1; // required
  string colorTheme = 2;
  string backgroundImageUrl = 3;
  string customDomain = 4;
}

A FileDescriptorSet can be generated for this file using the protocol buffer compiler.

protoc application_state.proto \
    --descriptor_set_out=application_state.pb \
    --include_imports

This descriptor set file can then be used in kplay's config file, alongside the descriptorName "sample.ApplicationState".

Read more about self describing protocol messages here.

By default, kplay operates under the assumption that brokers do not authenticate requests. Besides this, it supports AWS IAM authentication.

In case you get the kplay binary directly from a release, you may want to verify its authenticity. Checksums are applied to all released artifacts, and the resulting checksum file is signed using cosign.

Steps to verify (replace A.B.C in the commands listed below with the version you want):

1. Download the following files from the release:

   • kplay_A.B.C_checksums.txt
   • kplay_A.B.C_checksums.txt.pem
   • kplay_A.B.C_checksums.txt.sig

2. Verify the signature:

   cosign verify-blob kplay_A.B.C_checksums.txt \
       --certificate kplay_A.B.C_checksums.txt.pem \
       --signature kplay_A.B.C_checksums.txt.sig \
       --certificate-identity-regexp 'https://github\.com/dhth/kplay/\.github/workflows/.+' \
       --certificate-oidc-issuer "https://token.actions.githubusercontent.com"

3. Download the compressed archive you want, and validate its checksum:

   curl -sSLO https://github.com/dhth/kplay/releases/download/vA.B.C/kplay_A.B.C_linux_amd64.tar.gz
   sha256sum --ignore-missing -c kplay_A.B.C_checksums.txt

4. If checksum validation goes through, uncompress the archive:

   tar -xzf kplay_A.B.C_linux_amd64.tar.gz
   ./kplay
   # profit!

kplay is built using the awesome TUI framework bubbletea.