How to containerize RabbitMQ and deploy it on Cloud Run

[bhardwaju]

Introduction to containerization, Docker, and Cloud Run

Containerization is a technique that packages an application and its dependencies into a single unit, known as a container. Containers provide consistency across different environments, making it easier to develop, test, and deploy applications that run reliably regardless of the underlying infrastructure. 

Docker, a leading containerization platform, introduced the concept of Docker images, which serve as the building blocks for containers. Docker images are snapshots of a specific file system, along with the application code, runtime, libraries, and settings required to run the application. These images are created from a Dockerfile, a text file that outlines the instructions for building the image. By encapsulating everything needed for the application to run, Docker images eliminate compatibility issues and create a standardized environment. 

Cloud Run is a serverless compute platform that brings containerization and serverless concepts together. As a fully managed service, Cloud Run enables developers to deploy containerized applications without managing the underlying infrastructure. Cloud Run abstracts away the complexities of scaling, load balancing, and resource provisioning, so developers can focus solely on the application code. You simply deploy your containerized application to Cloud Run and Cloud Run will take care of the rest. 

Cloud Run is highly scalable to handle any amount of traffic, and provides a number of features that make it easy to develop, deploy, and manage your applications, including: 

• Automatic scaling: Cloud Run automatically scales your application up and down based on traffic.

• Zero-configuration networking: Cloud Run provides a secure and reliable network connection for your application.

• Automatic deployment: Cloud Run automatically deploys your application when you push a new container image to Cloud Build.

• Integrated logging and monitoring: Cloud Run provides integrated logging and monitoring for your application.

Introduction to RabbitMQ

RabbitMQ, an open-source message broker software that acts as an intermediary to facilitate communication between different applications or services. It allows decoupling of producers and consumers, enabling them to interact asynchronously and ensuring efficient data flow.

RabbitMQ stands as a cornerstone in modern application architecture, offering reliable and efficient communication between services, applications, and components. Its versatility, scalability, and support for various messaging patterns make it a popular tool for building robust and responsive distributed systems.

Key components of RabbitMQ

• Message queues: Message queues are the core of RabbitMQ. Messages are sent by producers to a queue, and consumers retrieve and process messages from the queue.
• Exchanges: Exchanges are responsible for routing messages from producers to queues based on predefined rules called bindings.
• Routing key: Routing keys determine how messages are routed from an exchange to a queue. They play a crucial role in message routing.
• Consumers: Consumers are applications or services that receive and process messages from queues.

Create a Docker image that publishes a message to RabbitMQ

Create Dockerfile 

Create a Dockerfile in a directory on your machine.

Choose base image 

Choose a base image for your Docker image. For simplicity, you can use an image with a programming language and libraries installed. For example, if you're using Python, you can use the python image.

Dockerfile

FROM python:3.9
Install Dependencies:
Install any dependencies your publishing script might need. If you're using a programming language, you can use the package manager for that language.

Dockerfile

RUN pip install pika
Copy Publishing Script:
Copy the script that publishes the message to RabbitMQ into the Docker image. Make sure the script is in the same directory as your Dockerfile.

Dockerfile

COPY publish.py .
Set Working Directory:
Set the working directory to the directory containing your script.

Here's an example publish.py script that uses the pika library to publish a message to a RabbitMQ queue.

Remember to replace the placeholders (RABBITMQ_HOST, RABBITMQ_PORT, RABBITMQ_USERNAME, RABBITMQ_PASSWORD, RABBITMQ_QUEUE, and MESSAGE_BODY) with your actual RabbitMQ connection details and message content.

import pika

# RabbitMQ connection details
RABBITMQ_HOST = 'your_rabbitmq_host'
RABBITMQ_PORT = 5672
RABBITMQ_USERNAME = 'your_username'
RABBITMQ_PASSWORD = 'your_password'
RABBITMQ_QUEUE = 'your_queue'

# Create a connection to RabbitMQ
credentials = pika.PlainCredentials(RABBITMQ_USERNAME, RABBITMQ_PASSWORD)
connection_params = pika.ConnectionParameters(host=RABBITMQ_HOST, port=RABBITMQ_PORT, credentials=credentials)
connection = pika.BlockingConnection(connection_params)

# Create a channel
channel = connection.channel()

# Declare a queue
channel.queue_declare(queue=RABBITMQ_QUEUE)

# Message to publish
MESSAGE_BODY = 'Hello, RabbitMQ!'

# Publish the message
channel.basic_publish(exchange='', routing_key=RABBITMQ_QUEUE, body=MESSAGE_BODY)

print(f" [x] Sent '{MESSAGE_BODY}'")

# Close the connection
connection.close()

Make sure to install the pika library before running the script using the following command: 

pip install pika

This script establishes a connection to the specified RabbitMQ server, declares a queue, and publishes a message to it. Replace the placeholders with the appropriate values based on your RabbitMQ setup.

Dockerfile

WORKDIR /usr/src/app
Command to Run Script:
Specify the command to run your publishing script. Modify the command according to your script's name and language.

Dockerfile

CMD ["python", "publish.py"]
Build Docker Image:
Open a terminal and navigate to the directory containing your Dockerfile. Run the following command to build the Docker image.

bash

docker build -t rabbitmq-publisher .
This command will build an image named rabbitmq-publisher.

Run Docker container 

After the image is built, you can run a Docker container from it. Make sure to pass any necessary environment variables or configuration options.

bash

docker run -e RABBITMQ_HOST=your_rabbitmq_host -e RABBITMQ_QUEUE=your_queue rabbitmq-publisher
Replace your_rabbitmq_host with the actual RabbitMQ host and your_queue with the queue name you want to publish the message to.

Remember to replace placeholders in the Dockerfile and commands with actual details based on your use case and chosen programming language. This example assumes you have a script named publish.py that utilizes the pika library for RabbitMQ interaction.

Push the Docker image

Once authenticated, you can push the Docker image to Google Container Registry using the following command: 

docker push gcr.io/PROJECT_ID/IMAGE_NAME[:TAG]
Replace PROJECT_ID and IMAGE_NAME with the appropriate values.

Wait for completion: The Docker image push might take some time depending on the image size and your network speed. Wait for the process to complete.

Verify the push

Navigate to Container Registry: Go to the Google Cloud Console, select your project, and go to "Container Registry" under "Tools."

Confirm image upload: You should see your pushed Docker image listed. Verify that the image has been successfully uploaded to the registry.

Deploying the containerized application on Cloud Run

Step 1: Deploy from Container Registry

Go to the Google Cloud Console: Log in to your Google Cloud Platform (GCP) account and access the Google Cloud Console.

Select your project: Choose the GCP project where you want to deploy your containerized RabbitMQ queue application.

Open Cloud Run: In the navigation menu, select "Cloud Run" under the "Compute" section.

Create a new service: Click on the "Create Service" button to initiate the deployment process.

Configure service settings: Fill in the necessary information, including the service name, deployment platform (Cloud Run), region, and authentication settings.

Deploy one revision from: Choose "Container Registry" as the source.

Select the container image: Choose the Docker image you previously pushed to Google Container Registry.

Advanced settings: Configure advanced settings, such as the number of allocated CPU and memory resources. You can also set the maximum number of container instances to allow for scaling.

Step 2: Access control and networking

Authentication and authorization: Cloud Run provides options to control who can access your service. You can choose between allowing all users or only authenticated users to access the service.

Networking settings: Configure networking settings to define how your service will handle incoming requests. You can choose between allowing unauthenticated invocations or requiring authentication.

Custom domains: If you have a custom domain, you can map it to your Cloud Run service. This enhances the user experience by using your own domain instead of the default Cloud Run URL.

Step 3: Monitor and scale

Cloud Monitoring: Once your containerized application is deployed, it's crucial to monitor its performance and health. Google Cloud Monitoring provides insights into various metrics, such as CPU utilization, memory usage, request latency, and error rates.

Set up alerts: Configure alerts to be notified when specific metrics exceed or near predefined thresholds. This proactive approach helps you address issues before they impact users.

Automatic scaling: One of the remarkable features of Cloud Run is its automatic scaling capability. As the incoming request load increases, Cloud Run automatically scales the number of container instances to handle the load. This ensures optimal performance and responsiveness.

Conclusion

By elaborating on the deployment steps for hosting a containerized RabbitMQ queue application on Cloud Run, we've provided you with a comprehensive guide to navigate through the intricacies of cloud deployment.

Containerization and serverless computing come together to offer a powerful platform that frees you from infrastructure management, allowing you to focus on your application's functionality.

---

Authored by:

Shivank Awasthi, Strategic Cloud Engineer (@shivankawasthi) 

Utkarsh Bhardwaj, Cloud Migration Consultant (@bhardwaju)