A full ecosystem that is built around kafka powered by golang.
Hope it can help you.
+-----------+
| VirtualIP |
+-----------+
|
+--------------+ Alert SOS Dashboard
| | | | | gk
+-------------------------------------------------------------------------------------------+
| | | | | | |
| +----------+ +----------+ | | | |
| | ehaproxy | | ehaproxy | | V | |
| +----------+ +----------+ | | | |
| | | | discovery | | | |
| +----------------+ | +-------------+ |
| | LB | | |
| | | +--------------------+ +--------+ |
| | +---| | election | | watch |
| |keepalive | zookeeper ensemble |-----------| kguard |-------------+ |
| | +---| | | | aggragator | |
| | | +--------------------+ +--------+ | |
| | +-------+ | | |
| | | registry | orchestration | | +- Pub
| +---------------+ |-----------+ +---------+ | | REST |
| | | | | | kateway |--|----|------|
| +---------+ +---------+ +--------+ +--------+ +---------+ | | |
| | kateway | | kateway | | actord | | actord | | | +- Sub
| +---------+ +---------+ +--------+ +--------+ | |
| | hh | | hh | | executor | |
| +----+ +----+ +--------------+ | |
| | | | | |
| | +---------+ +---------+ push | |
| +--------+ | JobTube | | Webhook |------------>-----------------|----|---Endpoints
| | | +---------+ +---------+ | |
| | | | scheduler | sub | |
| auth | |job WAL | dispatch | | |
| +------+ +---------------------------+------------------+ | |
| | | tenant shard | pubsub | flush | |
| +----------+ +---------+ +-------+ +------+ | |
| | auth DB | | DB Farm | | kafka | | TSDB | | |
| +----------+ +---------+ +-------+ +------+ | |
| | | | | | |
| | +----------------------------------------------+ | |
| | | | |
| | +-------------------------------------------+ |
| | |
| | zone |
+-------------------------------------------------------------------------------------------+
|
WebConsole
-
Elastic haproxy that sits in front of kateway.
-
A fully-managed real-time secure and reliable RESTful Cloud Pub/Sub streaming message/job service.
-
kateway job scheduler and webhook dispatcher.
-
Unified multi-datacenter multi-cluster kafka swiss-knife management console.
-
A handy zookeeper CLI that supports recursive operation without any dependency.
-
Kafka clusters body guard that emits health info to InfluxDB and exports key warnings to zabbix for alarting.
export PATH=$PATH:$GOPATH/bin
#========================================
# install go-bindata and go annotations
#========================================
go install github.com/jteeuwen/go-bindata/go-bindata
go install github.com/funkygao/goannotation
#========================================
# install gafka
#========================================
go get github.com/funkygao/gafka
cd $GOPATH/src/github.com/funkygao/gafka
./build.sh -a # build all components
#========================================
# try the gafka command 'gk'
#========================================
gk -h
The original project documentation reported the following production footprint, but the raw benchmark and operational artifacts are not present in this repository snapshot. Treat these numbers as historical context, not current production-readiness evidence:
- 4 data centers
- 50+ kafka clusters
- 100+ kafka brokers
- 500+ kafka topics
- 2000+ kafka partitions
- 10Billion messages per day
- peak transfer: 8TB per hour
Current, reproducible performance evidence is kept under docs/performance/ and includes the test environment and ACK boundary for each result.