Hands-on lab for inspecting KRaft metadata, diagnosing ISR state, simulating broker failure, and tuning min.insync.replicas.
3-node Apache Kafka 4.0 cluster, each node acting as both broker and controller (KRaft combined mode). No ZooKeeper.
kafka-1 → localhost:9092
kafka-2 → localhost:9093
kafka-3 → localhost:9094
Kafka UI → http://localhost:8090
- Docker & Docker Compose
- 4+ GB RAM available to Docker
Lab 0 — Start the cluster and verify quorum
Lab 1 — Inspect the @metadata log with kafka-dump-log.sh
Lab 2 — Create a topic (RF=3, 3 partitions) and produce messages
Lab 3 — Diagnose ISR state and replication lag
Lab 4 — Kill kafka-1, watch leader re-election
Lab 5 — Tune min.insync.replicas (durability vs availability)
Lab 6 — Recover kafka-1, observe ISR restoration
chmod +x scripts/*.sh
bash scripts/lab0-start-cluster.sh
bash scripts/lab1-inspect-metadata.sh
bash scripts/lab2-create-topic-produce.sh
bash scripts/lab3-check-isr-lag.sh
bash scripts/lab4-kill-broker.sh
bash scripts/lab5-tune-min-insync.sh
bash scripts/lab6-recover.sh| Concept | What to look for |
|---|---|
| KRaft quorum | kafka-metadata-quorum.sh describe --status → LeaderId, HighWatermark |
| @metadata log | kafka-dump-log.sh on __cluster_metadata-0/*.log → record types |
| ISR | kafka-topics.sh --describe → Isr: field |
| Replication lag | kafka-log-dirs.sh → offsetLag per partition/broker |
| Under-replicated | kafka-topics.sh --under-replicated-partitions |
| min.insync.replicas | kafka-configs.sh --alter → observe produce success/failure |
docker compose down -v # -v removes named volumes (log data)