CLIENT SERVER ARCHITECTURE
Before addressing the topic “Python and RabbitMQ” let’s talk a little about architectures by introducing the client-server model and distributed architectures.
THE CLIENT
Client software is generally of limited complexity, normally limited to operating as an interface to the server. In general, in the IT field, the term client indicates a component that accesses the services or resources of another component, called a server. In this context we can therefore speak of clients referring to hardware or software. A computer connected to a server via a local or geographic network, and to which it requests one or more services, using one or more network protocols is an example of a hardware client. An e-mail program is an example of a software client.
SOFTWARE DIVIDED INTO A CLIENT PART AND A SERVER PART
More and more software, such as the web, e-mail, databases, which are divided into a client part (residing and running on the client PC) and a server part (residing and running on the server). The term client also refers to the software used on the client computer to access the functionality offered by the server. For example, on the web, the client software is the web browser, and it talks to a web server through the HTTP protocol; for e-mail the client is called mail user agent or MUA (for example, Outlook, Mozilla Thunderbird, Eudora, …), and talks to the server (Mail Transfer Agent or MTA) through the SMTP protocol and POP or IMAP; the client for consulting or modifying the database (often made up of software libraries used by an application) talks to the DBMS, which manages the database and responds to client queries.
SERVER
The server software, in addition to the logical management of the system, must implement all the techniques of access management, allocation and release of resources, sharing and security of data or resources. For example, an e-mail server is comparable to any post office. Users to access their e-mail box via client must have been authorized. Similarly, a user must have the key to the mailbox located at a post office from which he wants to withdraw correspondence.
STRUCTURE OF A WEB APPLICATION
The figure shows the structure of a web application in which the client is a Browser that connects to the local or remote server. A web server runs on the server. In an architecture like this simplifying, there is a first phase in which the client generates the Request that it sends to the server. The server processes it if necessary, accesses a database to provide the requested data and returns it to the client in the form of a Response. The answer could be a web page, a message in Json format, Xml or an image etc.
DISTRIBUTED ARCHITECTURE
In distributed architectures there is a Producer who produces a certain information and applications that are interested in consuming the data produced and which are called Consumers. The Producer application notifies its information when it becomes available through a message that contains the information which can be textual, in json, xml format etc.
There are three essential characteristics to be able to speak of distributed architecture which are:
- When the application produces a message, the consumer is not necessarily available, in other words the application that must receive the message may not be active when it is produced. This does not mean that the message must be lost.
- Second consideration, the consumer could be more than one, that is, the same message may have to be delivered to a certain number of Consumers, even a very high number, not necessarily all available when the message is published.
- The Producer must never be aware of who and how many Consumers are, in order to produce a strong decoupling between producers and consumers.
PYTHON AND RABBITMQ – THE MESSAGE BROKER
Precisely to decouple Producer and Consumer, a third application is introduced which is interposed between the two in such a way as to produce a strong decoupling. The message broker is a server application, therefore installed on a server that is not normally the server that contains the Producer, nor the server that contains the Consumers. It has the task of acting as an intermediary between Producers and Consumers. It can be configured to keep messages indefinitely, so if a Consumer is not connected at a certain time, it can still receive the message from the message broker when it connects. A message broker can be configured with very sophisticated routing rules, to route the message also to different consumers.
There are various types of message brokers on the market, surely the most famous is RabbitMQ an open source product and this is the one we will use for our sample application. It has drivers for a large number of programming languages including Python.
PYTHON AND RABBITMQ – INSTALLATION OF RABBITMQ
Go to the address www.rabbitmq.com
- Select the Get Started tab and then Download + Installation.
- Choose the version for your operating system.
- Run Rabbit.
WE INSTALL PIKA THE PYTHON BOOKCASE FOR RABBIT
Open the RabbitMQ site again and select the Download + Installation. Go to the Client Library.
To install pika we will use the command line. For the terminal to recognize the pip3 command, it is necessary to register the Scripts path, located in a subdirectory of the folder where Python10 is installed, in the system variable Path. Reboot the system. If you want to be sure that the pika installation was successful, open the terminal and import the pika package.
RABBITMQ ARCHITECTURE OF A MESSAGE BROKER
From the consumer’s point of view, Rabbit provides us with the possibility of creating queues. Queues are like a sort of mailbox where messages are deposited and then forwarded to the Consumer (s) when the message becomes available. For each Consumer it is necessary to establish which queue it will use in order to use the messages sent by the Broker when available. At the opposite extreme, that is, putting ourselves on the side of the Producer are the Exchanges. They are the entry point to Rabbit and allow Producers to deliver their messages. Finally, there are the Binding, that is the rules configured within the message Broker to connect the Exchanges with the Queues. Through the Binding, the Producers and Consumers are completely decoupled.
CONNECTION TO RABBITMQ
The first thing is to establish a Connection, once the connection is established we can request a communication channel that through the connection allows us to reach the Message Broker. A single connection can host multiple communication channels. Multiple applications can use the same connection and different channels.
Normally this configuration is used to distribute the workload of a possibly huge series of messages to n consumers all connected to the same queue. Processing takes place in parallel. By connecting all consumers to a single queue, Rabbit will route the first message to the first consumer, the second message to the second consumer, the third to the third and so on, always repeating the cycle. This mechanism is called Round-Robin and allows for extremely scalable applications as the Consumers work in parallel.
import pika
print('Collegamento a RabbitMQ......')
params = pika.ConnectionParameters(host='localhost')#PARAMETERS TO PASS TO THE CONNECTION
connections=pika.BlockingConnection(params)#CONNECTION TO RABBITMQ
channel = connections.channel() #GET THE CHANNEL
channel.queue_declare(queue='worker_queue')#CODA THAT CONSUMERS OR WORKERS WILL USE
print('Eseguito......')
#WE SEND TO THE BROKER 100,000 MESSAGES IN QUICK SEUQENCE
i=0
while True:
message = str(i)
i+=1
#WE MUST PUBLISH THE MESSAGE ON THE EXCHANGE
#THE routing_key IS THE DEFINITION OF BINDING
#WE ASK THE EXCHANGE TO PUBLISH ALL THAT
#THE ALREADY DEFINED worker_queue ARRIVES ON A QUEUE
channel.basic_publish(exchange='',routing_key='worker_queue',body=message)
print(f'Inviato messaggio {message}')
if i > 100_000:
break
#CLOSING THE CONNECTION TO RABBITMQ
connections.close()
import pika
#THE CONNECTION PHASE IS THE SAME VIEW WITH THE PRODUCER
print('Collegamento a RabbitMQ......')
params = pika.ConnectionParameters(host='localhost')
connections=pika.BlockingConnection(params)
channel = connections.channel()
channel.queue_declare(queue='worker_queue')
print('Eseguito......')
#WE MUST INDICATE TO RABBIT THROUGH PIKA WHAT WE MUST DO EVERY
#TIME WE RECEIVE A MESSAGE. TO DO THIS WE WRITE A FUNCTION
#DI CALLBACK THAT WILL BE PERFORMED EVERY TIME A
#MESSAGE FROM THE TAIL.
def callback(channel,method,properties,body):
print(f'Ricevuto messaggio {body}')
channel.basic_consume(queue='worker_queue',on_message_callback=callback,auto_ack=False)
#start_consuming() CAUSES THE ARREST OF THE WORKER WHICH WILL
#WAITING FOR THE PUBLICATION OF MESSAGES FROM THE PRODUCER.
channel.start_consuming()
PYTHON AND RABBITMQ – LINK TO GITHUB CODE
PYTHON AND RABBITMQ – LINK TO PREVIOUS POST
To run the programs, open the terminal and set the folder containing the consumer.py and producer.py programs as the current directory. Open four tabs in the terminal, out of three run the consumer.py programs using the following command: python consumer.py wait for the Consumers to wait. On the fourth terminal, run producer.py. View the execution on the 4 tabs.
Leave A Comment