Week 6
Lecture
Message-oriented middleware --> started in stock exchange broker in 1970s
Cool to demo for project : Publisher is python based, the subscriber is javascript based, they communicate with AMQP
BTL: A connection pool is a cache (pool) of database or network connections that can be reused rather than creating a new connection every time. It is commonly used with databases, message brokers (like RabbitMQ), and HTTP clients to improve performance and reduce overhead.
DONT DRAW THE AMQP BROKER AS BOX
Message Oriented Middleware (MOM)
A soft ware which sits between two or more applications or services and allow them to exchange data in the form of messages
The connection overall forms hub-spoke architecture patterns, in contrast to peer-to-peer connections in invocation-based communication technologies
Message Structure
AMQP Message sample
Networking / communication technology related info:
<BlockingChannel impl=<Channel number=1 OPEN conn=<SelectConnection OPEN transport=<pika.adapters.utils.io_services_utils._AsyncPlaintextTransport object at 0x03A700D0> params=<ConnectionParameters host=localhost port=5672 virtual_host=/ ssl=False>>>><br><Basic.Deliver(['consumer_tag=ctag1.8d2bc570099a4128b1abc19953315247', 'delivery_tag=1', 'exchange=order_topic', 'redelivered=False', 'routing_key=order.info'])>
Properties / metadata:
<BasicProperties(['correlation_id=8c7cf2b5-4199-4ddf-98ba-c3b43fa9f8a3', 'delivery_mode=2', 'reply_to=none'])>
Body / business data:
{"customer_id": "Apple TAN", "order_id": 10, "timestamp": "2020-01-02 00:44:27.021964", "order_item": [{"book_id": "9781434474234", "quantity": 1, "item_id": 1, "order_id": 10}, {"book_id": "9781449474212", "quantity": 1, "item_id": 2, "order_id": 10}]}
Advanced Message Queuing Protocol (AMQP)
An open standard protocol for MOM
Define a message format that allows standard and extensible representations of various types of data
(In theory) Provide interoperability among different AMQP-compliant implementation software that can be used on either the client side or the server side
RabbitMQ
A message-queuing software (message broker/queue manager) that supports AMQP (and other protocols)
It accepts and forwards messages
Message can be data or event
It keeps the message in a queue until a consumer takes the message off the queue (storage for message)
Multiple queues can be created; a single message can be sent to multiple queue
Message is routed via an exchange and key pattern matching
one queue can send out messages to one subscriber; for load balancing
Durable and Persistent
A queue or exchange may or may not be durable. If not durable, when reboots happen or no subscriber when teh message reach the queue, the message sent to queue may be lost and exchange would be gone.
Persistent message: ( delivery_mode=2 or pika.spec.PERSISTENT_DELIVERY_MODE );
Not persistent message: (i.e., delivery_mode=1 or pika.spec.TRANSIENT_DELIVERY_MODE ), the message would be dropped if there is no receiver online when the message is sent.
Type of exchanges (key pattern matching)
Exchange: sth that JSQ doesnt have
The sender/publisher doesn't need to know any receiving queue's name or other existence, as long as it knows the right routing key to use.
A key is a sequence of words separated by ".", e.g., order.info, this.is.also.a.syntactically.valid.key
The exchange in the broker routes the message to the queue(s) with the matching binding key.
A receiver consumes the message(s) routed to the queue(s) that it is consuming.
1) Direct
exchange delivers a message to a binding key of queue is exactly the same as routing key of message
2) Topic
exchange delivers a message to binding keys of queues that match with the wildcard of the routing pattern of exchange
* match only one wildcard word
# match with any number of wildcard word
3) Fanout
exchange delivers a message to queues that are bound to the exchange without utilising binding key and routing key
Pika Programming Model (Python)
Publisher
1. Create connection
# create a physical network connection to the broker
connection = pika.BlockingConnection(…)
2. Create Channel
# create a virtual lightweight connection (session) to the broker for an application/microservice
channel = connection.channel(…)
3. Declare Exchange
# create an exchange if it doesn't exist in the broker
channel.exchange_declare(…)
4. Prepare + Publish Message Content (Publisher)
channel.basic_publish(…) # publish message; input is routing key
5. Close connection
connection.close()
Subscriber
1. Create connection
# create a physical network connection to the broker
connection = pika.BlockingConnection(…)
2. Create Channel
# create a virtual lightweight connection (session) to the broker for an application/microservice
channel = connection.channel(…)
3. Declare Exchange
# create an exchange if it doesn't exist in the broker
channel.exchange_declare(…)
4. Declare Queue
# create a queue if it doesn't exist in the broker
channel.queue_declare(…)
5. Bind Queue
# bind a queue to an exchange in the broker
channel.queue_bind(…) #input is binding key
6. Consume + Process Messages from Queue (Subscriber)
channel.basic_consume(…) # sets up a consumer and binds the “on_message_callback” function to all messages to be received.
channel.start_consuming() # starts a loop to wait to receive any message from the queue, and automatically invokes the "on_message_callback" function to process each of the messages received. Use Ctrl + C in the cmd windows to terminate it.
7. Close connection
connection.close()
Load Balancing
In reality, the queue may use a random message distribution strategy, instead of a strict round-robin mode. Then, each message may go to a random receiver.
Since each queue can only send 1 message to one container, this mechanism creates load balancing.
