A from-scratch Kafka-compatible broker built as part of the CodeCrafters "Build Your Own Kafka" challenge. Implements the Kafka binary wire protocol over TCP and supports the core Produce / Fetch flow with on-disk persistence.
| API | Key | Versions | What it does |
|---|---|---|---|
| Produce | 0 | 0 – 11 | Validates topic/partition, writes RecordBatch to disk |
| Fetch | 1 | 0 – 16 | Reads RecordBatch bytes from partition log file |
| ApiVersions | 18 | 0 – 4 | Advertises supported APIs and version ranges |
| DescribeTopicPartitions | 75 | 0 – 0 | Returns topic/partition metadata from cluster log |
cmd/broker/
main.go entry point — reads server.properties, wires everything together
internal/
protocol/
protocol.go Kafka binary protocol: API keys, error codes, request header,
response builder, COMPACT_STRING / COMPACT_ARRAY helpers
metadata/
metadata.go Parses __cluster_metadata-0 log file;
builds in-memory TopicMetadata + PartitionMetadata maps
handler/
handler.go HandlerFunc type + Registry (maps API key → handler)
apiversions.go ApiVersions handler
describetopics.go DescribeTopicPartitions handler
fetch.go Fetch handler
produce.go Produce handler
server/
server.go TCP listener, framed message reading, API key dispatch
Client TCP connection
│
│ [4-byte big-endian message size] + [payload bytes]
▼
server.handleConn
│ reads size → reads payload → parses RequestHeader
│ (api_key int16, api_version int16, correlation_id int32)
▼
handler.Registry.Get(apiKey) → HandlerFunc
│ routes to the registered handler
▼
HandlerFunc(header, payload) → []byte
│ parses request body, builds binary response
▼
conn.Write(response)
Data root is the log.dirs value from server.properties (default /tmp/kraft-combined-logs).
<log.dirs>/
__cluster_metadata-0/
00000000000000000000.log ← TopicRecord + PartitionRecord batches
<topic>-<partition>/
00000000000000000000.log ← RecordBatch data for that partition
When a Produce request arrives for a valid topic/partition, the raw RecordBatch bytes from the request are appended directly to the partition log file (create + mkdir if needed).
- Every message is framed:
[INT32 size][payload] - Flexible API versions (v9+) use compact encodings:
COMPACT_STRING—uint8(len+1)then bytesCOMPACT_ARRAY—uint8(count+1)then elementsTAG_BUFFER— trailing0x00on every struct
- Request header v2: fixed 8 bytes +
NULLABLE_STRINGclient_id + TAG_BUFFER - Response header v1: correlation_id + TAG_BUFFER
# Build
go build -o /tmp/broker ./cmd/broker
# Run
/tmp/broker /tmp/server.propertiesMinimal server.properties:
log.dirs=/tmp/kraft-combined-logsThe broker listens on port 9092.
To submit to CodeCrafters:
git push origin master