mq-bench — Messaging Transport Benchmarking (MQTT + Redis + NATS + Zenoh + RabbitMQ/AMQP)

Asynchronous messaging is a cornerstone of modern distributed systems, enabling decoupled communication for scalable and resilient applications. Today's message broker ecosystem spans a wide range of designs — from high-throughput streaming platforms to lightweight protocols tailored for edge and IoT environments — yet choosing the right broker remains difficult. Existing evaluations largely focus on throughput and latency on fixed hardware while overlooking CPU and memory footprint under resource constraints, factors that are critical for edge and IoT deployments.

mq-bench is a unified benchmarking framework for evaluating message brokers under identical conditions. It provides a comprehensive, scriptable harness for pub/sub and request/reply across eight broker implementations: Mosquitto, EMQX, HiveMQ, RabbitMQ, ActiveMQ Artemis, NATS Server, Redis (Pub/Sub), and Zenoh Router. The tool scales experiments up to 10,000 concurrent client pairs across multiple VM configurations representative of edge hardware, collecting latency (P50/P95/P99), throughput, CPU, and memory metrics at each scale point.

Built in Rust with Tokio. Ships a Docker Compose stack with MQTT (Mosquitto/EMQX/HiveMQ), Redis, NATS, RabbitMQ, ActiveMQ Artemis, and an optional Zenoh 3-router star. The harness uses a pluggable transport abstraction with adapters for MQTT, Redis, NATS, Zenoh (1.5.1), and RabbitMQ (AMQP via lapin). Select an engine via --engine and pass connection options via --connect KEY=VALUE.

The research findings from this project have been accepted at a peer-reviewed venue:

Tapajit Chandra Paul, Pawissanutt Lertpongrujikorn, Hai Duc Nguyen, and Mohsen Amini Salehi, "Benchmarking Message Brokers for IoT Edge Computing: A Comprehensive Performance Study," in Proceedings of the 26th IEEE International Symposium on Cluster, Cloud, and Internet Computing (CCGrid 2026), 2026.

The paper covers latency vs. payload size, throughput and resource utilization under client scaling (1-to-1 topology, three VM configurations), and reliability under network disruptions (QoS evaluation with TCP RST fault injection) for eight message broker implementations. Results and raw data are available in the final_results/ directory.

One extended result that could not be completely included in the paper due to space constraints — a detailed fanout topology analysis — is available here:
Fanout Topology Analysis: Throughput, CPU, and Memory

mq-bench/
├── src/                  # Rust source — CLI, transport adapters, metrics, roles
├── tests/                # Integration and smoke tests
├── scripts/              # Experiment automation scripts (run_baseline.sh, run_fanout.sh, lib.sh, ...)
├── setup/                # Environment setup scripts and KVM/Docker configuration guides
├── config/               # Broker configuration files (Zenoh routers, MQTT configs)
├── final_results/        # All benchmark results, plots, and analysis
│   ├── latency/          # Latency vs. payload size experiment results
│   ├── throughput/       # Throughput & resource utilization results (vm12/, vm24/, vm48/, fanout/)
│   ├── latex/            # LaTeX source for the paper's evaluation section
│   └── fanout_analysis.md  # Extended fanout topology analysis (see above)
├── artifacts/            # Raw per-run output artifacts (auto-generated during experiments)
├── docker-compose.yml    # Broker service stack
└── Cargo.toml

Setup Guide — Instructions for provisioning KVM VMs on bare-metal nodes, configuring macvtap networking, and installing Docker for multi-node benchmark deployments.

• Docker Compose stack: MQTT (Mosquitto/EMQX/HiveMQ), Redis, NATS, RabbitMQ, ActiveMQ Artemis, plus an optional Zenoh 3-router star (router2 → router1 ← router3)
• Roles: pub, sub, req, qry; plus multi-topic mt-pub/mt-sub
• Open-loop publisher with rate control
• Requester with QPS pacing, concurrency, and timeouts
• Subscriber with latency measurement (ns timestamps embedded in payload)
• CSV snapshots to stdout or file (--csv)

Performance-focused implementation details

• Handler-based subscribe and query registration (lower overhead than streams)
• Reusable declared publisher for hot-path sends
• Payload header is fixed 24 bytes; subscribers decode for E2E latency
• Batching and CSV flushing for live tailing

Transports (current)

• mqtt, redis, nats, zenoh, rabbitmq (AMQP)
  • rabbitmq: native AMQP adapter (alias: amqp). Connect via --connect url=amqp://user:pass@host:port/%2f.
  • rabbitmq-mqtt: use RabbitMQ’s MQTT plugin through the MQTT adapter (host: 127.0.0.1, port: 1886 by default).
  • artemis: alias to MQTT adapter to hit Artemis’s MQTT listener (127.0.0.1:1887). You can also use mqtt-artemis transport.
  • Broker-specific MQTT aliases are available: mqtt-mosquitto, mqtt-emqx, mqtt-hivemq, mqtt-rabbitmq, mqtt-artemis.
  • Wildcards: use slash style in CLI (e.g., bench/**). Adapters map to engine-native patterns (e.g., NATS a.>; AMQP uses amq.topic with dot routing internally).
  • Topics: use slash form in CLI (e.g., bench/topic); adapters map as needed (e.g., NATS and AMQP use dots under the hood).

1. Bring up services (MQTT/Redis/NATS; Zenoh routers optional)

# From repo root
docker compose up -d
# Verify ports (bound to 127.0.0.1)
# Redis: 6379 | NATS: 4222 | MQTT: 1883/1884/1885 | RabbitMQ: AMQP 5672, MQTT 1886 | Artemis: AMQP 5673, MQTT 1887 | Zenoh: 7447/7448/7449 (optional)

2. Build the harness

cargo build --release

3. Run quick pub/sub sanity tests

MQTT (defaults to Mosquitto on 1883)

./target/release/mq-bench sub --engine mqtt --connect host=127.0.0.1 --connect port=1883 --expr bench/topic
./target/release/mq-bench pub --engine mqtt --connect host=127.0.0.1 --connect port=1883 --topic-prefix bench/topic --payload 200 --rate 5 --duration 10

Redis

./target/release/mq-bench sub --engine redis --connect url=redis://127.0.0.1:6379 --expr bench/topic
./target/release/mq-bench pub --engine redis --connect url=redis://127.0.0.1:6379 --topic-prefix bench/topic --payload 200 --rate 5 --duration 10

NATS (default port 4222)

./target/release/mq-bench sub --engine nats --connect host=127.0.0.1 --connect port=4222 --expr bench/topic
./target/release/mq-bench pub --engine nats --connect host=127.0.0.1 --connect port=4222 --topic-prefix bench/topic --payload 200 --rate 5 --duration 10

Zenoh (optional cross-router: sub on router2/7448, pub to router3/7449)

./target/release/mq-bench sub --endpoint tcp/127.0.0.1:7448 --expr bench/topic
./target/release/mq-bench pub --endpoint tcp/127.0.0.1:7449 --payload 200 --rate 5 --duration 10

You should see subscriber logs like:

DEBUG: Received message seq=12, latency=1.10ms
Subscriber stats - Received: 20, Errors: 0, Rate: 4.00 msg/s, P99 Latency: 3.01ms

Top-level:

• --run-id STRING tag outputs/CSV (optional)
• --out-dir PATH base artifacts directory (default ./artifacts)
• --log-level trace|debug|info|warn|error (default info)
• --snapshot-interval SECS stats snapshot cadence (default 1)

All roles accept a transport engine and connection options:

• --engine zenoh|mqtt|redis|nats|rabbitmq (alias: amqp)

• --connect KEY=VALUE (repeatable)

• Zenoh back-compat: --endpoint tcp/host:port maps to --connect endpoint=...

• Publisher (pub)

  • Topic/key selection
    • --topic-prefix base key (defaults to bench/topic)
    • Multi-topic: --topics N and --publishers M to create M logical publishers over N topics
  • --payload size in bytes
    • --rate messages per second (omitted or <= 0 means unlimited)
    • --duration seconds
  • --csv path/to/pub.csv to write CSV snapshots to a file (stdout if omitted)

• Subscriber (sub)

  • --expr key expression, e.g. bench/topic or bench/**
  • --csv path/to/sub.csv to write CSV snapshots to a file (stdout if omitted)

• Requester (req)

  • --key-expr query key expression
  • --qps queries per second (omitted or <= 0 means unlimited)
  • --concurrency max in-flight
  • --timeout per-query timeout (ms)
  • --duration seconds
  • --csv path/to/req.csv

• Queryable (qry)

  • --serve-prefix repeatable; prefixes to serve
  • --reply-size reply body size in bytes
  • --proc-delay processing delay per query (ms)
  • --csv path/to/qry.csv

• Multi-topic publisher (mt-pub)

  • --topic-prefix base key (e.g., bench/topic)
  • Dimensions: --tenants T --regions R --services S --shards K
  • --publishers M logical publishers (<= TRS*K; -1 uses total keys)
  • --mapping mdim|hash key mapping across publishers
  • --payload, --rate, --duration, --csv, --share-transport

• Multi-topic subscriber (mt-sub)

  • --topic-prefix base key (e.g., bench/mtopic)
  • Dimensions: --tenants T --regions R --services S --shards K
  • --subscribers N number of per-key subscriptions (<= total keys; -1 uses total keys)
  • --mapping mdim|hash, --duration, --csv, --share-transport

Tip: If you pass --csv ./artifacts/run1/pub.csv or sub.csv, parent directories will be created automatically.

Engine connect hints:

• MQTT: --connect host=127.0.0.1 --connect port=1883
  • Optional auth: add --connect username=USER --connect password=PASS
• Redis: --connect url=redis://127.0.0.1:6379
• NATS: --connect host=127.0.0.1 --connect port=4222
• Zenoh: --endpoint tcp/127.0.0.1:7448 (or --connect endpoint=tcp/127.0.0.1:7448)
• RabbitMQ (AMQP): --connect url=amqp://guest:guest@127.0.0.1:5672/%2f
• RabbitMQ (MQTT plugin): use MQTT engine at host 127.0.0.1 port 1886
• Artemis (MQTT): use MQTT engine at host 127.0.0.1 port 1887

With services up and the binary built:

1. Start a queryable on Zenoh router2 (optional, 7448):

./target/release/mq-bench qry --endpoint tcp/127.0.0.1:7448 --serve-prefix bench/topic --reply-size 256 --proc-delay 0

2. Run a requester on Zenoh router3 (7449):

./target/release/mq-bench req --endpoint tcp/127.0.0.1:7449 --key-expr bench/topic --qps 1000 --concurrency 32 --timeout 2000 --duration 5

NATS

# Start a responder (subscribe and reply)
./target/release/mq-bench qry --engine nats --connect host=127.0.0.1 --connect port=4222 \
  --serve-prefix bench/topic --reply-size 256 --proc-delay 0

# Run a requester
./target/release/mq-bench req --engine nats --connect host=127.0.0.1 --connect port=4222 \
  --key-expr bench/topic --qps 1000 --concurrency 32 --timeout 2000 --duration 5

Redis (simple list-based req/rep baseline)

./target/release/mq-bench qry --engine redis --connect url=redis://127.0.0.1:6379 \
  --serve-prefix bench/topic --reply-size 256 --proc-delay 0

./target/release/mq-bench req --engine redis --connect url=redis://127.0.0.1:6379 \
  --key-expr bench/topic --qps 1000 --concurrency 32 --timeout 2000 --duration 5

Notes:

• Start the queryable before the requester.
• Add --csv path/to/req.csv or qry.csv to save snapshots.
• RabbitMQ AMQP: request/reply is not implemented yet in the adapter; pub/sub is supported.

Publish over many keys and subscribe to all of them in one process:

# Subscriber over many keys
./target/release/mq-bench mt-sub --engine mqtt --connect host=127.0.0.1 --connect port=1883 \
  --topic-prefix bench/mtopic --tenants 4 --regions 2 --services 3 --shards 8 \
  --subscribers -1 --duration 10

# Matching publisher
./target/release/mq-bench mt-pub --engine mqtt --connect host=127.0.0.1 --connect port=1883 \
  --topic-prefix bench/mtopic --tenants 4 --regions 2 --services 3 --shards 8 \
  --publishers -1 --payload 256 --rate 1000 --duration 10

Docker Compose defines optional services (all bound to 127.0.0.1):

• MQTT brokers: Mosquitto 1883, EMQX 1884, HiveMQ 1885, RabbitMQ MQTT 1886, Artemis MQTT 1887
• RabbitMQ: AMQP 5672
• Artemis: AMQP 5673
• Redis: 6379
• NATS: 4222
• Zenoh star (optional): 7447; 7448→7447; 7449→7447

Configs in config/ have discovery disabled to ensure deterministic static peering.

• Generic, multi-transport:
  • scripts/run_baseline.sh (ENGINE=zenoh|mqtt|redis|nats|rabbitmq)
  • scripts/run_fanout.sh (ENGINE=zenoh|mqtt|redis|nats|rabbitmq)
• Convenience wrappers:
  • Baseline: run_mqtt_baseline.sh, run_redis_baseline.sh, run_nats_baseline.sh, run_rabbitmq_baseline.sh, run_artemis_baseline.sh
  • Fanout: run_mqtt_fanout.sh, run_redis_fanout.sh, run_nats_fanout.sh, run_rabbitmq_fanout.sh, run_artemis_fanout.sh

Examples for RabbitMQ and Artemis:

RabbitMQ (AMQP)

./target/release/mq-bench sub --engine rabbitmq --connect url=amqp://guest:guest@127.0.0.1:5672/%2f --expr bench/topic
./target/release/mq-bench pub --engine rabbitmq --connect url=amqp://guest:guest@127.0.0.1:5672/%2f --topic-prefix bench/topic --payload 200 --rate 5 --duration 10

RabbitMQ (MQTT plugin)

./target/release/mq-bench sub --engine mqtt --connect host=127.0.0.1 --connect port=1886 --expr bench/topic
./target/release/mq-bench pub --engine mqtt --connect host=127.0.0.1 --connect port=1886 --topic-prefix bench/topic --payload 200 --rate 5 --duration 10

Artemis (MQTT)

./target/release/mq-bench sub --engine mqtt --connect host=127.0.0.1 --connect port=1887 --expr bench/topic
./target/release/mq-bench pub --engine mqtt --connect host=127.0.0.1 --connect port=1887 --topic-prefix bench/topic --payload 200 --rate 5 --duration 10

Minimal steps to introduce a new engine (e.g., "foo"):

• Adapter: create src/transport/foo.rs that implements Transport (must support subscribe and create_publisher; request/register_queryable can return NotImplemented initially). Parse --connect KEY=VALUE from ConnectOptions.params and pass payload bytes through unchanged.
• Wire-up in code:
  • src/transport/mod.rs: add Engine::Foo and call foo::connect(...) under a #[cfg(feature = "transport-foo")] gate.
  • src/transport/config.rs: map engine strings (e.g., "foo") in parse_engine to Engine::Foo.
  • Cargo.toml: add feature transport-foo enabling the adapter’s deps.
• Scripts: in scripts/lib.sh, teach make_connect_args how to turn ENGINE=foo into --engine foo --connect ... pairs; optionally add cases in scripts/orchestrate_benchmarks.sh for baseline/fanout labels.
• Docker (optional): add a service to docker-compose.yml and bind ports to 127.0.0.1.
• Docs: list the engine under “Transports (current)” with example connect hints.